JP2020129287A - Process information acquisition system, process information acquisition method, and process information acquisition program - Google Patents

Abstract

To provide a process information acquisition system capable of acquiring information on facility, worker status, and articles without using a plurality of sensors or an electronic tag carried by a worker.SOLUTION: A process information acquisition system 1 includes: a signal light 51 of a facility 50; a camera 20 that includes a working range of a worker in the shooting range; and a calculation unit that acquires image data from the camera 20, recognizes an operating state of the facility 50 and a working state of the worker from an image in the image data, recognizes articles within the shooting range from the image in the image data, and collects the operating state of the facility 50 as facility information, the working state of the worker as worker information, and the type and status of articles as article information.SELECTED DRAWING: Figure 1

Description

The present invention relates to a process information acquisition system, a process information acquisition method, and a process information acquisition program.

2. Description of the Related Art In recent years, in factories using equipment such as automatic machine tools, an analysis system has been developed for recognizing the operating status of equipment and the working state of workers and improving productivity.

For example, in Patent Document 1, position information is received from an RFID tag given to a worker and facility operation information is received from an electric signal of a production facility, and the work ratio of the worker is automatically calculated. Disclosed is a work analysis system in which an operator can save the labor of observing the work of an operator by photographing and recording the work state of the worker.

This work analysis system includes an information receiving device that receives an information signal emitted from an RFID (Radio Frequency Identifier) tag, a host system A that processes information of the information receiving device as position information, and an electric signal transmitted from a production facility. An electric signal receiving device for receiving the signal, a host system B for processing the electric signal of the electric signal receiving device, a camera for photographing the working state of the worker, a recorder for recording and reproducing the image photographed by the camera, and a camera photographing The host system C that controls the recording and reproduction of the recorder, the network device that connects the camera, the recorder, and the host system C, the position information from the host system A, the equipment operation information from the host system B, and the operator. A database A that stores the RFID tag correspondence information and equipment position information, an analysis device that receives the correspondence information, position information, and operation information from the database A and analyzes the work of the worker, and stores the output that is analyzed by the analysis device. It has a database B and a monitoring terminal for visualizing the output stored in the database B.

JP 2012-203770 A

In the related art, a signal of an RFID tag carried by a worker (person) is received by a dedicated information receiving device, an electric signal from equipment is received by another dedicated electric signal receiving device, and an image from a camera is further received. Is received and recorded by a dedicated recorder. Further, in the conventional technique, signals or images received or recorded by the plurality of dedicated devices are further collected by different upper systems A, B, and C, and these higher systems A, B, and C are networked. By connecting by, multiple signals and images collected are analyzed.

For this reason, in the related art, various sensors for collecting information on facilities and people and electronic tags carried by workers such as RFID tags are required. In addition, in the related art, the information on the article regarding the production is not obtained. For this reason, in the conventional technology, a large investment is required to include from the selection of sensors and electronic tags necessary for collecting equipment and human information to construction. In addition, it is necessary for the worker to carry the electronic tag without fail, which imposes an additional man-hour on the worker. Further, in the related art, since the time series of collected information is different, there are various problems such as a lot of man-hours for integrating these.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide equipment, worker status, and article information without using a plurality of sensors or an electronic tag carried by a worker. A process information acquisition system, a process information acquisition method, and a process information acquisition program that can be acquired.

The above object of the present invention is achieved by the following means.

(1) Equipment and a camera including the working range of the worker in the shooting range,
Acquiring image data from the camera, recognizing an operating state of the equipment and a working state of the worker from the image of the image data, and recognizing an article within a shooting range from the image of the image data. A computing unit that collects the operating state of the facility as facility information, the working state of the worker as worker information, and the type and state of the article as article information;
And a process information acquisition system.

(2) The arithmetic unit recognizes the operating state of the equipment from the change in the color of the signal lamp or the change in the light emitting position reflected in the image of the image data, and the operator's image of the operator in the image of the image data is recognized. The process information acquisition system according to (1) above, which recognizes a work state of the worker from a position and/or a direction.

(3) The camera includes a wide-angle lens or a fisheye lens,
The process information acquisition system according to (1) or (2) above, which includes an image correction unit that corrects distortion when an image captured using a wide-angle lens or a fisheye lens has distortion.

(4) Having a storage unit for storing the image data,
The process information acquisition system according to any one of (1) to (3) above, wherein the arithmetic unit stores the image data for a predetermined period in the storage unit.

(5) The computing unit calculates a resource spent for producing a product by the facility and/or a lead time required for production from the collected facility information, the worker information, and the article information. The process information acquisition system according to any one of (1) to (4).

(6) A step (a) of acquiring image data from a camera for photographing the work range of the equipment and the worker,
(B) recognizing an operating state of the equipment and a working state of the worker from the image of the image data, and recognizing an article in the image of the image data;
Collecting the operating status of the equipment as equipment information, the work status of the worker as worker information, and the type and status of the article as article information;
And a process information acquisition method.

(7) In the step (a), the operation state of the equipment is recognized based on the change in the color of the signal lamp or the change in the light emitting position shown in the image of the image data, and the operation shown in the image of the image data is performed. The process information acquisition method according to (6) above, wherein the work state of the worker is recognized from the position and/or orientation of the worker.

(8) The camera includes a wide-angle lens or a fisheye lens,
Between the step (a) and the step (b), there is a step (d) for correcting a distortion in an image taken using the wide-angle lens or the fisheye lens, which is (6). Alternatively, the process information acquisition method according to (7).

(9) The process information acquisition method according to any one of (6) to (8), further including a step (e) of storing the image data in a storage unit in a predetermined period.

(10) Furthermore, a step (f) of calculating a resource spent for production of a product by the equipment and/or a lead time required for production from the collected equipment information, worker information, and article information. The method for acquiring process information according to any one of (6) to (9) above.

(11) A process information acquisition program for causing a computer to execute the process information acquisition method according to any one of (6) to (10).

According to the present invention, the operating state of equipment, the working state of a worker, and the recognition of an article are recognized from an image captured by a camera, so that a plurality of sensors and an electronic tag carried by the worker are not used. .. As a result, initial investment can be suppressed, there is no burden on the operator, there is no need to time-series data, and it is possible to obtain information about the equipment, the operator, and the article.

It is explanatory drawing for demonstrating the structure of the process information acquisition system of embodiment. It is a block diagram for explaining the functional composition of the process information acquisition system of an embodiment. It is explanatory drawing for demonstrating detection of the position and direction of a worker using a skeleton recognition technique. It is a flow chart which shows a processing procedure.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same elements will be denoted by the same reference symbols, without redundant description. Further, the dimensional ratios in the drawings are exaggerated for convenience of description, and may differ from the actual ratios.

FIG. 1 is an explanatory diagram for explaining the configuration of the process information acquisition system of the embodiment. FIG. 2 is a block diagram for explaining the functional configuration of the process information acquisition system of the embodiment. The equipment, the work table, and the size, orientation, and arrangement of the workers illustrated in FIG. 1 are conceptual only for explaining the present embodiment, and the actual work site is as described above. It does not have such a structure.

The process information acquisition system 1 includes a server 10 and a camera 20 connected to the server 10.

The camera 20 includes a signal light 51 of the facility 50 and a work range of a worker as a shooting range. In the present embodiment, the imaging range of the camera 20 includes the signal lamp 51 of the equipment 50, the operation panel 52 of the equipment 50, the door 53, and the work table 55. The work range of the worker includes these and is a range necessary for the worker to work on them.

The equipment 50 may be any equipment, for example, an automatic processing machine such as a numerical control processing machine. Further, in the following description, production includes processing of an article by an automatic processing machine.

The camera 20, for example, including the equipment 50 and the work table 55, is installed at a position where the range in which the worker works is photographed from above. By photographing from above the worker, the direction in which the worker is extending his/her arm is recognized in the photographed image. Therefore, in the present embodiment, the orientation of the worker is recognized from the direction in which the worker extends his arm (details will be described later). Furthermore, it is more preferable that the camera 20 is installed at a position where the operation panel 52 of the facility 50 can be photographed. In particular, the camera 20 is more preferably installed at a position where the display on the operation panel 52 can be photographed. For these reasons, in the present embodiment, the camera 20 is installed diagonally above the equipment 50 so that the operator can be photographed from above and the operation panel 52 can also be photographed. From the content displayed on the operation panel 52, in addition to the information of the signal lamp 51, the state of the equipment 50 can be recognized in more detail.

The camera 20 is not particularly limited as long as it can capture a moving image or continuous still images. Specifically, as the camera 20, for example, a general movie camera or a still camera can be used. The camera 20 has an interface (not shown) for communicating with the server 10, as described later. Image data is output from the camera 20 and transmitted to the server 10. The image data is data indicating a moving image or continuous still images, but in the present embodiment, a moving image is taken. When shooting a series of still images, the shooting interval is set to a time interval at which the work state can be recognized. If this time interval is too wide, it becomes difficult to recognize the work state. Although it depends on the work content, for example, it is preferable to set the shooting interval to 10 seconds or less.

In addition, the camera 20 is preferably provided with, for example, a wide-angle lens or a fish-eye lens so that one camera 20 can photograph the entire work range. When a wide-angle lens or a fisheye lens is provided, the captured image may be distorted. When a wide-angle lens or a fish-eye lens is provided, it is preferable to provide an image correction unit for correcting image distortion peculiar to the wide-angle lens or the fish-eye lens, separately from the camera 20 side or the server 10 side. When a wide-angle lens or a fish-eye lens is provided, image processing can be efficiently performed by using the image data that has been subjected to image correction by the image correction unit. In the present embodiment, the server 10 has the function of the image correction unit 115 (see FIG. 2). The function of the image correction unit 115 can be realized by software.

As shown in FIG. 2, the server 10 has a calculation unit 11, a storage unit 12, and an interface 13 (I/F).

The calculation unit 11 has, as its functions, an equipment state recognition unit 111 that recognizes the operating state of the equipment 50 from image data, a work recognition unit 112 that recognizes the work state of a worker, and an article recognition unit that recognizes the type and state of an article. 113, an analysis processing unit 114 that analyzes the operating state of the entire process from the operating state of the facility 50, the working state of the worker, and the type and state of the article. Details of the function of each unit will be described later. Further, the calculation unit 11 has an image correction unit 115 as needed. The function of the image correction unit 115 is as described above.

The arithmetic unit 11 is a processor (CPU: Central Processing Unit or GPU: Graphics Processing Unit) that executes various arithmetic processes in the server 10.

The storage unit 12 is a storage device included in the server 10, and is configured by appropriately combining, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), and an HDD (Hard Disk Drive). The ROM is a read-only storage device that stores various programs and various data. RAM is a high-speed random access storage device that temporarily stores programs and data as a work area. The HDD is a large-capacity random access storage device that stores various programs and various data. Of course, the storage unit 12 may be another storage device.

The interface 13 receives input of image data from the camera 20. The interface 13 may be, for example, a wired communication interface such as Ethernet (registered trademark), SATA, PCI Express, or USB in accordance with the interface 13 provided in the camera 20, Bluetooth (registered trademark) based on a standard such as IEEE1394, IEEE802. 11 is a wireless communication interface. The interface 13 included in the server 10 is used not only for receiving an input from the camera 20 but also for transmitting/receiving data to/from another external device. The image data input to the server 10 is described as digital data in the present embodiment, but is not limited to this. If the image data output by the camera 20 is analog data, a device for converting the image data into digital data may be provided in the camera 20, the server 10, or between the camera 20 and the server 10. When the server 10 receives analog data, the server 10 includes an interface 13 for analog lines.

The server 10 is a general computer, and each function is achieved by the computer serving as the server 10 executing a program for executing the function of each unit.

Hereinafter, the function of the server 10 will be further described.

The calculation unit 11 stores the received image data in the storage unit 12. The period to be stored is selected as needed, for example, several hours to several days, several weeks, or the like. Of course, the storage unit 12 secures a capacity necessary for storing the image data.

The equipment state recognition unit 111 recognizes the color of the signal lamp 51 provided in the equipment 50 from the image represented by the image data, and recognizes the operating state of the equipment 50.

The equipment state recognition unit 111 reads out the image data almost at the same time as the image data is stored in the storage unit 12 and processes it in real time. By being processed in real time, the current operating state is determined.

The signal lamp 51 is also referred to as a laminated signal lamp or a signal tower, and has, for example, three-color lamps of green, yellow, and red that emit light according to the operating state of the facility 50. Each color of the signal light 51 indicates the following states, for example. Green indicates that the equipment 50 is operating normally. Yellow indicates a normal suspension status. The red color indicates the status of abnormal stop due to occurrence of trouble. Further, when the signal lamp 51 is turned off, it indicates that the equipment 50 is normally stopped. In the working process, when the article to be processed is installed in the facility 50 or taken out from the facility 50, the facility 50 is normally powered off for safety. In such a case, the signal lamp 51 is turned off. In addition, depending on the facility 50, there is a facility in which articles can be installed and removed without turning off the power. In the case of such a facility 50, the signal light 51 turns on or blinks yellow when installing or taking out an article. Further, the signal lamp 51 is turned off even when the facility 50 is intentionally stopped. The emission color of the signal lamp 51 and the state of the equipment 50 at that time can be arbitrarily set, and are not limited to the above combinations.

The equipment state recognition unit 111 presets a predetermined area in which the signal lamp 51 appears in the image, and recognizes the operating state of the equipment 50 from the color change in the predetermined area. The equipment state recognition unit 111 may recognize the operating state of the equipment 50 from the light emitting position instead of the color of the signal lamp 51. The signal lights 51 are, for example, in the order of red, yellow, and green from the top.

The equipment state recognition unit 111 can also recognize the operating state of the equipment 50 from the display on the operation panel 52. In this case, the equipment state recognition unit 111 sets a region in the image where the operation panel 52 is shown in advance, and recognizes the character in the region. In the present embodiment, since the camera 20 and the equipment 50 are both fixed objects, the area where the operation panel 52 appears does not move even if the image is a moving image. For example, when the equipment 50 is an automatic processing machine, the characters displayed on the operation panel 52 are not so many types. The character displayed on the operation panel 52 may be, for example, time or a code number indicating the current state. These are often alphanumeric abbreviations or numbers. Therefore, the equipment state recognition unit 111 may only be able to recognize alphanumeric characters. Of course, the equipment state recognition unit 111 may recognize not only alphanumeric characters but also hiragana, katakana, kanji, etc. from an image.

For example, when the signal light 51 is green and a number is displayed on the operation panel 52, the number is recognized as the time until the end of the work or the elapsed time from the start of the work. It may be set in advance whether the time to the work end or the elapsed time, or if it is determined that the time to the work end when the numerical value decreases with time, the elapsed time when it increases Good. Further, when the signal lamp 51 is yellow or red and a number is displayed on the operation panel 52, it is recognized that a code indicating the reason for the stop is displayed. For example, the code displayed when the signal light 51 is yellow may be recognized as a stop for confirmation of the equipment 50 during processing, a stop due to the end of processing, or the like. If the signal lamp 51 is red, it is recognized that a code indicating an abnormality is displayed. The storage unit 12 stores the numerical values of these codes and the reasons corresponding to them. The equipment state recognition unit 111 determines the reason for stopping by comparing the read code with the reason stored in the storage unit 12. The determined stop reason can be notified to the process manager of the stop of the facility 50 and the reason thereof by being transmitted to, for example, a computer for process management (not shown).

Further, for example, in an automatic machine tool, a voltage value, a current value, and other numerical values are displayed on the operation panel 52. In such a facility 50, the state of the machining currently being executed is finely recognized as the operating state of the facility 50 by recognizing the display (numerical value) of the voltage value and the current value as characters.

As another aspect, for example, the image showing the operation panel 52 may be simply transmitted to a computer for process control or the like. In this case, the characters may not be recognized from the display on the operation panel 52. Further, in this case, the area of the operation panel 52 may be set in advance so as to be enlarged and displayed on the computer for process control.

Furthermore, the equipment state recognition unit 111 can recognize an input operation on the operation panel 52 by monitoring the operation panel 52.

As described above, in the present embodiment, not only the color of the signal lamp 51 but also the display of the operation panel 52 can be read to recognize a more detailed operating state of the facility 50.

The equipment state recognition unit 111 can also recognize the operating state of the equipment 50 by recognizing the open/closed state of the door 53 from the position of the door 53 of the equipment 50. The open/closed state of the door 53 is recognized, for example, by extracting the characteristic of the door 53 from the image and the position of the characteristic portion is different between the frames of the images arranged in time series. The feature of the door 53 is a feature amount for identifying the door 53 from the image. Such a feature amount may be, for example, the entire door 53 or an accessory (for example, a handle) that moves together with the door 53. The feature amount is stored in the storage unit 12 in advance and used when identifying the door 53 from the image.

Specifically, for example, if the door 53 is open, it is recognized that an article has been taken in and out of the facility 50. As a combination with the signal light 51, for example, if the light is turned off before the door 53 is opened, it is recognized that the article is installed in the facility 50 when the door 53 is opened. Further, it is recognized that the article is taken out of the facility 50 when the door 53 is green until the door 53 is opened and then turned off, and the door 53 is opened. Further, when the door 53 is opened when it is yellow, it is recognized that the article being processed is confirmed.

As described above, in the present embodiment, by further recognizing the open/closed state of the door 53, the operating state of the facility 50 can be recognized more finely.

In the present embodiment, information on the equipment 50 obtained by the equipment state recognition unit 111, that is, the emission color (including extinction) of the signal lamp 51, the character string or operation recognized from the operation panel 52, and the open/closed state of the door 53 are displayed. Collectively referred to as equipment information.

Next, the work recognition unit 112 detects the position and orientation of the worker from the image data, and recognizes whether the work is for the equipment 50 (equipment work) or another work (non-equipment work).

An existing technique can be used to detect the position and orientation of the worker from the image data. Examples of existing technologies include open pose (Open Pose (see https://github.com/CMU-Perceptual-Computing-Lab/openpose)) and deep pose (Deep Pose (https://www.slideshare). .net/mitmul/deeppose-human-pose-estimation-via-deep-neural-networks))) can be used. The open pose and deep pose are techniques for recognizing a posture by estimating a human skeleton from two-dimensional image data. Such a technique is called a skeleton recognition technique. Besides, a technique using a depth camera (RBG-D camera) or TOF (Time of Flight) can be used. In this case, as the camera 20, instead of a normal movie camera, a dedicated camera capable of image recognition including the distance may be used.

The detection of the position and orientation of the worker using the skeleton recognition technology will be described with specific examples. FIG. 3 is an explanatory diagram for explaining the detection of the position and orientation of the worker using the skeleton recognition technology.

The shooting range in FIG. 3 is set so as to overlap the area in which the worker works. Therefore, the worker works within the shooting range. The range in which the worker works is called the work range. Instead of this, the photographing range may be set wider than the work range, and the work recognition unit 112 may detect only the workers within the work range preset in the image of the image data. ..

First, the work recognition unit 112 detects a worker within the shooting range. Then, when the work recognizing unit 112 detects the worker, the work recognizing unit 112 acquires the coordinate value of the main skeleton such as the head or shoulder of the worker. The position of the worker is detected from the coordinate values of the main skeleton. In the skeleton recognition technique, a coordinate system required for processing is set in advance in the shooting range. For example, a two-dimensional coordinate system is set, but a three-dimensional coordinate system may be set instead. Further, the position of the worker may be determined from the coordinate values of the skeleton of the foot as well as the head and shoulders.

Next, the work recognition unit 112 determines the direction in which the arm (particularly the part beyond the elbow) projects with respect to the main skeleton such as the head and shoulders, and determines that direction in the direction in which the worker is facing. Judge that there is. In the present embodiment, the orientation of the worker is determined from the orientation of at least one arm. This may be one-handed work, although both arms of the worker may project toward the equipment 50 during work on the equipment 50 or the like. Specifically, the one-handed work includes, for example, changing the setting of the program and restarting. Such work can be performed with one hand on the operation panel 52 of the facility 50. At this time, the worker is lowering the other arm along his body. In such a case, one arm is projecting toward the equipment 50, but the other arm is not projecting. Therefore, in the present embodiment, as described above, the orientation of the worker is determined from the direction in which at least one arm is facing, and the working state of the worker is further determined from that.

In FIG. 3, for example, the worker 30 a can determine that the operator 50 is facing the facility 50 because the direction in which one arm is protruding is the direction of the facility 50. Therefore, the work recognition unit 112 determines that the worker 30a is working on the facility 50.

Further, the worker 30b can determine that the worker 30b faces the work table 55 because the direction in which both arms are projected is the direction of the work table 55. Therefore, the work recognition unit 112 determines that the worker 30b is working on the work table 55.

Furthermore, the worker 30c does not have the direction in which both arms are projected, not the direction of the equipment 50, nor the direction of the work table 55. In such a case, the work recognition unit 112 determines that the worker 30c is not working in the shooting range, that is, the work range.

In the present embodiment, the recognition result of the position and orientation of the worker obtained by the work recognition unit 112 is referred to as worker information.

Next, the article recognition unit 113 recognizes the type and state of the article from the image. The recognition of the article is executed by image processing, for example. The article recognition unit 113 recognizes the shape of an object in the image by image analysis and compares it with the characteristics (feature amount) of a predetermined article stored in advance. As a result of comparison, when the difference in characteristics is within the predetermined range, the article recognition unit 113 recognizes the object recognized in the image as the same type of article as the predetermined article. In the recognition of this article, it takes time to process all the objects in the image. Therefore, in the present embodiment, the article is recognized in a predetermined area within the image. The predetermined area is, for example, on the work table 55, near the door 53 of the facility 50, or the like. As a result, the article recognition unit 113 recognizes the presence or absence of an article on the work table 55 as the state of the article, and if there is an article, recognizes the type. Further, the predetermined area may be in the vicinity of the conducting wire from the work table 55 to the facility 50 on which the worker moves. In this way, the recognition speed can be increased by deciding the predetermined area for the recognition of the article. Further, the characteristic (feature amount) stored as a predetermined article is not one or one type, but by storing the characteristic (feature amount) of various different types of articles, various types of different articles can be stored. Can be recognized.

As another method of recognizing the article, for example, a sticker attached to the article, a mark drawn on a tag attached to the article, an article code, or a bar code is read by the article recognizing unit 113. May be. The barcode may be one-dimensional or two-dimensional. It is preferable to predetermine the range of reading the sticker or tag from the image. The range in which the seal or tag is read is, for example, on the work table 55, in the vicinity of the door 53 of the facility 50, in the vicinity of the conductor of the worker, or the like.

In the present embodiment, the product recognition result obtained by the product recognition unit 113 is referred to as product information.

Next, the analysis processing unit 114, based on the equipment information collected by the equipment state recognition unit 111, the worker information collected by the work recognition unit 112, and the article information collected by the article recognition unit 113, the imaging range. Find the operating status of the entire process inside. The analysis processing unit 114 also performs various analyzes based on these pieces of information.

To determine the operating state of the entire process, for example, a predetermined determination table is used. The determination table is table data that describes a combination of facility information, worker information, and article information, and a work procedure corresponding to the combination.

Table 1 is an example of the determination table.

This will be described with reference to Table 1.

For example, in the procedure (1), the content of the worker information is the worker's recognition, the signal light 51 is off as the content of the facility information, the operation panel 52 and the door 53 are unrecognized, and the content of the article information is Items are recognized on the work table 55. At this stage, since the worker is recognized for the first time, the content of the worker information is the worker recognition. At this time, the number of workers is also specified. Further, the equipment 50 is stopped, and the article is first recognized on the work table 55. As a result, it is determined that the article to be processed is being selected.

Then, in the procedure (2), the contents of the worker information are equipment work, the contents of the equipment information are that the signal light 51 is off, the operation panel 52 is not recognized, and the door 53 is the recognition of door opening/closing. The content of the item information is unrecognized. At this stage, the article has disappeared from the work table 55, the door 53 of the equipment 50 is opened and closed, and the worker is working. Further, the equipment 50 is stopped. And the procedure is after the selection of the article. From these states, it is determined that the article is installed in the facility 50. As a result, it is determined that the article is installed. Note that the equipment work is, for example, the information that the worker is heading for the equipment 50 and is determined to be the work performed on the equipment 50 by the worker.

Then, in the procedure (3), the contents of the worker information are equipment work, and as the contents of the equipment information, the signal lamp 51 is turned off, the operation panel 52 recognizes the input operation, and the door 53 does not recognize, The content of the item information is unrecognized. At this stage, it is determined that the worker is operating the operation panel 52 after installing the article in the facility 50. Further, the equipment 50 is stopped. As a result, it is determined that the equipment is being operated.

Then, in the procedure (4), the contents of the worker information are the equipment work, the contents of the equipment information are green of the signal light 51, the operation panel 52 and the door 53 are unrecognized, and the contents of the article information are There is no recognition of the item. At this stage, it is determined that the equipment 50 has started operating normally. This determines that the equipment is operating.

Then, in the procedure (5), the contents of the worker information are equipment work or table work (equipment work/table work), the signal light 51 is green, and the operation panel 52 and the door 53 are not recognized as the contents of the equipment information. Therefore, the content of the article information indicates that the article is not recognized. At this stage, since the equipment 50 is operating normally, it is determined that the worker is performing other work. As a result, it is determined that the work is incidental. The other work is various, for example, measuring another article or checking the operating state of the equipment 50.

Subsequently, in step (6), the operator information content is equipment work, the equipment information content is yellow for the signal light 51, the operation panel 52 and the door 53 are unrecognized, and the content of the article information is There is no recognition of the item. At this stage, it is determined that the equipment 50 is normally stopped and the worker looks inside the equipment 50 to confirm the processing state of the article. As a result, it is determined that the item is confirmed.

Then, in the procedure (7), the operator information content is equipment work, the equipment information content is yellow for the signal light 51, the operation panel 52 for recognition of the input operation, and the door 53 for no recognition. The content of the item information indicates that the item is not recognized. At this stage, it is determined that the facility 50 is preparing for continuous operation. As a result, it is determined that the equipment is operated.

Subsequently, in step (8), the contents of the worker information are equipment work, the contents of the equipment information are green, the signal light 51 is green, the operation panel 52 and the door 53 are unrecognized, and the contents of the article information are There is no recognition of the item. At this stage, it is determined that the equipment 50 is operating normally. This determines that the equipment is operating.

Here, steps (6) to (8) may be repeated.

Then, in the procedure (9), the contents of the worker information are equipment work, the contents of the equipment information are that the signal light 51 is off, the operation panel 52 is not recognized, and the door 53 is the recognition of door opening/closing, The content of the item information indicates that the item is not recognized. At this stage, it is determined that the equipment 50 is normally stopped and the door 53 is opened and closed. Moreover, since the processing has been completed and stopped from the flow of the procedure, it is judged that the article is taken out. As a result, it is determined that the article has been removed.

Then, in the procedure (10), the content of the worker information is a table work, the content of the equipment information is that the signal light 51 is off, the operation panel 52 and the door 53 are unrecognized, and the content of the article information is The article is recognized on the work table 55. At this stage, it is determined that the worker is working on the work table 55. Moreover, it is judged that the article is being measured because the processing has been completed from the flow of the procedure. As a result, it is determined that the item is measured.

Then, in the procedure (11), the content of the worker information is equipment work, the content of the equipment information is that the signal lamp 51 is off, the operation panel 52 and the door 53 are unrecognized, and the content of the article information is It is unrecognized. At this stage, the worker is working on the equipment. Moreover, since it is after the end of the measurement of the article from the flow of the procedure, it is determined that the article is washed in the facility 50. As a result, it is determined that the article is washed.

Subsequently, in step (12), the contents of the worker information are table work, the contents of the facility information are that the signal lamp 51 is off, the operation panel 52 and the door 53 are unrecognized, and the contents of the article information are The article is recognized on the work table 55. At this stage, it is determined that the worker is working on the work table 55. Moreover, since it is after the end of washing the articles from the flow of the procedure, it is judged here that the articles are inspected. As a result, it is determined to be an article inspection.

The determination table also describes determinations out of the procedure and combinations thereof.

For example, in the out-of-procedure 1, the content of the worker information is equipment work, the content of the equipment information is that the signal light 51 is turned off, the operation panel 52 is not recognized, and the door 53 is the recognition of opening and closing of the door. The content is unrecognized. When such a combination of information is recognized, if the signal lamp 51 is red in the stage before the out-of-procedure 1, it is determined that the repair is in progress.

Further, in the out-of-procedure 2, the content of the worker information is not recognized, the content of the equipment information is that the signal light 51 is red, the operation panel 52 is an error display recognition, and the door 53 is not recognition. The content is unrecognized. When such a combination of information is recognized, there is a possibility that no action is taken although the facility 50 is stopped for some reason. When such a combination of information is recognized, it is determined that the work is delayed.

The content of the determination table described above is merely an example for the purpose of description, and naturally, it varies depending on various processes and the content of processing of the article.

Next, an example of analysis of information by the analysis processing unit 114 will be described.

For example, the analysis processing unit 114 aggregates each information in one dashboard. The analysis processing unit 114 also calculates the resources spent for producing the product. The calculated resource is utilized for cost calculation, for example. Further, the analysis processing unit 114 calculates the lead time for producing the product and sets it as the actual value. The calculated actual value is utilized, for example, for predictive management for comparing the actual value with a budget or planned time set as a target, and for improving the accuracy of estimation when acquiring a project.

As a specific analysis, for example, the resource is calculated from the number of workers obtained in the selection of the article of the above-described work procedure (1) as the number of workers spent per facility. The resource may be the total number of workers who have spent the work procedures (1) to (12). Further, when the voltage value or the current value is read from the operation panel 52, the resource may be used as the resource of the facility 50 by obtaining the amount of electric power from those values.

Further, as the production lead time, the entire production time can be obtained from the time taken for the above-mentioned work procedures (1) to (12). Further, the actual time when the facility 50 is used can be obtained from the time taken for the work procedures (4) to (8).

In this embodiment, a plurality of pieces of information such as facility information, worker information, and article information are collected from one image data. Therefore, since each information is originally collected from one image data, the time series is the same. Therefore, when each information is aggregated in one dashboard, time series adjustment is unnecessary. The aggregated plurality of information is transmitted to, for example, a computer for process control. The process control computer uses the received information to perform further various analyses.

Next, a processing procedure of acquiring and analyzing each information will be described. FIG. 4 is a flowchart showing the processing procedure. In this processing procedure, the arithmetic unit 11 (specifically, the CPU) executes the program created based on this processing procedure, whereby the operation of each unit as the function of the arithmetic unit 11 described above is performed.

First, the calculation unit 11 receives and acquires image data from the camera 20 (S1). The acquired image data is stored in the storage unit 12.

Subsequently, the calculation unit 11 reads the stored image data and collects equipment information, worker information, and article information from the image (S2).

Subsequently, the arithmetic unit 11 makes each collected information into one so as to be easily analyzed, and stores this in the storage unit 12 (S3). If there is already aggregated information at the stage of S3, the calculation unit 11 updates the information.

After S3, if there is no input of the end of processing (S4: NO), the arithmetic unit 11 returns to S1 as it is, successively receives image data, and continues each step. When the processing end is input (S4: YES), the calculation unit 11 ends the processing.

According to this embodiment, the following effects are achieved.

In this embodiment, a plurality of pieces of information such as equipment information of the equipment 50, worker information, and article information are obtained from image data obtained by one camera 20. Therefore, in the present embodiment, since a plurality of pieces of information required for process control are originally acquired from one image data, the time series are the same. There is no need to match the series. Therefore, in the present embodiment, information processing is easier than in the related art, and the processing speed can be increased.

Further, in this embodiment, a plurality of pieces of information such as equipment information of the equipment 50, worker information, and article information are obtained from image data obtained by one camera 20. Therefore, in the present embodiment, unlike the prior art, without providing a sensor for detecting the operating state of the equipment 50 or using an electronic tag carried by the worker, the state of the equipment 50 or the worker, the state of the article Information can be obtained. Therefore, in the present embodiment, it is only necessary to install the camera 20, and the existing server 10 can be used for processing the image data. Alternatively, even if the server 10 is newly installed, only a general server computer needs to be installed. Therefore, the present embodiment does not require a separate host system that collects information from a plurality of sensors or electronic tags, unlike the related art. Therefore, the present embodiment requires less initial investment than the related art. As a result, the present embodiment can collect a plurality of pieces of information in the process while suppressing the initial investment, can be used for various analyzes such as process management and predictive management, and can be used for appropriate improvement actions.

Further, in the present embodiment, since the data for obtaining the plurality of information is one image data, it is not necessary to store the plurality of information as in the conventional technique. Therefore, this embodiment can reduce the amount of data. Further, in the present embodiment, since transmission/reception of one image data is sufficient, it is possible to reduce the load on the band related to the transmission/reception of data.

Although the embodiments of the present invention have been described above, the present invention can be variously modified.

In the embodiment, the example in which the server 10 is used for collecting and analyzing information has been described, but not limited to the server 10, for example, a small computer (such as a board PC) is incorporated in the camera 20 to be integrated with the camera 20. Then, the computer for executing the above-mentioned functions may be executed by this small computer. By doing so, the present invention can be implemented only by installing the camera 20.

Further, each function of the server 10 described above may be replaced by a cloud server. For this purpose, for example, the camera 20 is provided with an interface 13 that can be connected to the Internet, and the image data is directly transmitted to the cloud server. Then, by executing the processing already described on the cloud server, each information can be analyzed on the cloud server, and each information and the analysis result can be browsed from the terminal of the user at the remote location.

Further, in the embodiment, the area including one facility 50 is set as the image capturing range of the camera 20, but the image capturing range may be such that a plurality of the facility 50 can be imaged at one time. In this case, which equipment 50 the worker is working on is determined from the orientation of the worker who is heading to each equipment 50. Alternatively, a work range may be provided for each facility 50, and it may be determined that a worker within each work range is working on the facility 50.

Further, in the embodiment, the work state of the worker is determined from the position and the direction of the worker, but if a work determination area is provided around the facility 50 and the worker is within the work determination area, It may be determined that the facility work is being performed. On the contrary, the determination may be made only from the orientation of the worker. In particular, when the photographing range is narrow, for example, when the worker's moving range is about 5 steps or less, if the worker is facing the facility 50, the facility work is performed. If the worker is not facing the facility 50, other work is performed. Or you can judge that you are not working.

Further, in the embodiment, the three-color signal lamp 51 is used, but the present invention is not limited to three colors, and can be applied to a two-color signal lamp, one-color signal lamp, and the like. For example, in the case of a two-color signal light, there are normal operation in green and normal stop in yellow or abnormal stop in red. When the embodiment is made to correspond to such a facility 50, the processing for yellow or red may not be performed. Further, even in the case of one color, or in the case of four colors or five colors, the processing may be performed according to the operating state of the equipment 50.

In the embodiment, the input operation on the operation panel 52 and the opening/closing of the door 53 are recognized by the equipment state recognition unit 111 described as the function of the calculation unit 11. However, such a function is merely an example, and for example, in the work recognition unit 112, the operator extends his hand (fingertip) to the operation panel 52, and the operator extends the door 53 to open and close the door 53. The input operation to the operation panel 52 and the opening/closing of the door 53 may be recognized by the operation of the operator such as being present.

Further, in the embodiment, as the facility information, not only the signal lamp 51 but also the states of the operation panel 52 and the door 53 are recognized, but the operating state of the facility 50 may be recognized only from the signal lamp 51.

In addition, the conditions, numerical values, and the like used in the description of the embodiments are merely for explanation, and the present invention is not limited to these conditions and numerical values.

Further, the process management computer mentioned in the embodiment is a computer connected to the server 10, for example, a computer provided at a work site or a computer provided in a process management department located at a remote place. Further, the computer for process management may be a mobile terminal such as a tablet or a smartphone held by a process manager or an operator.

The program according to the present invention can also be realized by a dedicated hardware circuit. In addition, the program is provided by a computer-readable recording medium such as a USB (Universal Serial Bus) memory or a DVD (Digital Versatile Disc)-ROM (Read Only Memory), or a network such as the Internet is used regardless of the recording medium. It is also possible to provide it online via. In this case, the program is usually stored in a magnetic disk device or the like that constitutes the storage unit. In addition, the program can be provided as a single application software, or can be provided by incorporating it into another software as one function.

Furthermore, the present invention can be modified in various ways based on the configurations described in the claims, and these are also included in the scope of the present invention.

1 Process information acquisition system,
10 servers,
11 arithmetic unit,
12 memory,
13 interfaces,
20 cameras,
50 facilities,
51 signal lights,
52 Operation panel,
53 doors,
55 work table,
111 Equipment status recognition unit,
112 work recognition department,
113 article recognition unit,
114 analysis processing unit,
115 Image correction unit.

Claims (11)

A camera that includes the equipment and the working range of the worker in the shooting range,
Acquiring image data from the camera, recognizing an operating state of the equipment and a working state of the worker from the image of the image data, and recognizing an article within a shooting range from the image of the image data. A computing unit that collects the operating state of the facility as facility information, the working state of the worker as worker information, and the type and state of the article as article information;
And a process information acquisition system. The calculation unit recognizes the operating state of the equipment from the change in the color of the signal lamp or the change in the light emitting position shown in the image of the image data, and detects the position of the worker and/or the position of the worker shown in the image of the image data. The process information acquisition system according to claim 1, wherein the work status of the worker is recognized from the direction. The camera includes a wide-angle lens or a fisheye lens,
The process information acquisition system according to claim 1 or 2, further comprising an image correction unit for correcting, when an image captured by using a wide-angle lens or a fisheye lens has a distortion. A storage unit for storing the image data,
The process information acquisition system according to claim 1, wherein the arithmetic unit stores the image data for a predetermined period in the storage unit. The computing unit calculates a resource spent for producing a product by the facility and/or a lead time required for production from the collected facility information, the worker information, and the article information. 4. The process information acquisition system described in any one of 4. A step (a) of acquiring image data from a camera for photographing the working range of the facility and the worker,
(B) recognizing an operating state of the equipment and a working state of the worker from the image of the image data, and recognizing an article in the image of the image data;
Collecting the operating status of the equipment as equipment information, the work status of the worker as worker information, and the type and status of the article as article information;
And a process information acquisition method. In the step (a), the operating state of the equipment is recognized from the change in the color of the signal lamp or the change in the light emitting position shown in the image of the image data, and the position of the worker shown in the image of the image data is recognized. The process information acquisition method according to claim 6, wherein the work state of the worker is recognized from the direction and/or the direction. The camera includes a wide-angle lens or a fisheye lens,
7. The method according to claim 6, further comprising a step (d) between the step (a) and the step (b), for correcting a distortion of an image captured by using the wide-angle lens or the fisheye lens. 7. The process information acquisition method described in 7. The process information acquisition method according to claim 6, further comprising a step (e) of storing the image data of a predetermined period in a storage unit. Furthermore, there is a step (f) of calculating a resource spent for production of a product by the equipment and/or a lead time required for production from the collected equipment information, the worker information, and the article information. Item 10. The process information acquisition method according to any one of items 6 to 9. A process information acquisition program for causing a computer to execute the process information acquisition method according to claim 6.