JP7139987B2 - Process information acquisition system, process information acquisition method, and process information acquisition program - Google Patents

Description

The present invention relates to a process information acquisition system, a process information acquisition method, and a process information acquisition program.

2. Description of the Related Art In recent years, in factories using equipment such as automatic machine tools, analysis systems have been developed for recognizing the operating conditions of equipment and working conditions of workers, and for improving productivity.

For example, in Patent Document 1, location information is received from an RFID tag held by a worker, and equipment operation information is received from an electrical signal of a production equipment, and the work ratio of the worker is automatically determined, and together with a camera A work analysis system is disclosed that saves an observer the trouble of observing a worker's work by photographing and recording the work state of the worker.

This work analysis system includes an information receiver that receives information signals emitted from RFID (Radio Frequency Identifier) tags, a host system A that processes information from the information receiver as positional information, and electrical signals emitted from production equipment. an electrical signal receiving device that receives the electrical signal, a host system B that processes the electrical signal of the electrical signal receiving device, a camera that captures the working state of the worker, a recorder that records and reproduces the video captured by the camera, and the camera and the host system C that controls the recording and playback of the recorder, the network equipment that connects the camera, the recorder, and the host system C, the location information from the host system A, the equipment operation information from the host system B, and the worker A database A that stores RFID tag correspondence information and equipment position information, an analysis device that receives the correspondence information, position information, and operation information from the database A and analyzes the work of the worker, and stores the output analyzed by the analysis device. It has a database B and a monitoring terminal that visualizes the output stored in database B.

Japanese Unexamined Patent Application Publication No. 2012-203770

In the prior art, the signal of the RFID tag carried by the worker (person) is received by a dedicated information receiving device, the electrical signal from the facility is received by another dedicated electrical signal receiving device, and the image from the camera is received. is received and recorded by a dedicated recorder. Furthermore, in the prior art, the signals and images received or recorded by these multiple dedicated devices are further collected by separate host systems A, B, and C, and these host systems A, B, and C are networked. By connecting with each other, the multiple signals and images collected are analyzed.

For this reason, the prior art requires various sensors for collecting information on facilities and people, and electronic tags such as RFID tags that workers carry. Also, the prior art does not provide information about the production of the article. Therefore, in the conventional technology, a large investment is required from the selection of sensors and electronic tags necessary for collecting information on facilities and people to construction work. In addition, the worker must always carry an electronic tag, which imposes an additional man-hour on the worker. Furthermore, in the prior art, since the time series of the collected information is different, there are various problems such as the need for many man-hours to integrate the information.

The present invention has been made in view of the above circumstances, and an object of the present invention is to obtain information on facilities, worker status, and articles without using a plurality of sensors or electronic tags carried by workers. It is to provide a process information acquisition system, a process information acquisition method, and a process information acquisition program capable of acquiring.

The above objects of the present invention are achieved by the following means.

(1) a camera including equipment and a working range of a worker in its imaging range;
Acquiring image data from the camera, recognizing the operating state of the equipment and the working state of the worker from the image of the image data, and recognizing an article within a shooting range from the image of the image data , the operation status of the facility is collected as equipment information, the work status of the worker as worker information, and the type and state of the article as article information, and based on the facility information, the worker information, and the article information, a calculation unit that determines a work procedure for a process within the imaging range ;
A process information acquisition system.

(2) The computing unit recognizes the operating state of the facility from a change in color or a change in the light emitting position of the signal lamp shown in the image of the image data, and recognizes the operation status of the worker shown in the image of the image data The process information acquisition system according to (1) above, wherein the work state of the worker is recognized from the position and/or orientation.

(3) the camera has a wide-angle lens or a fisheye lens;
The process information acquisition system according to (1) or (2) above, which has an image corrector for correcting distortion in an image captured using a wide-angle lens or a fisheye lens.

(4) having a storage unit for storing the image data;
The process information acquisition system according to any one of (1) to (3) above, wherein the calculation unit stores the image data for a predetermined period in the storage unit.

(5) The computing unit calculates the resources used to produce the product by the equipment and/or the lead time required for production from the collected equipment information, worker information, and article information. The process information acquisition system according to any one of (1) to (4).

(6) step (a) of acquiring image data from a camera that captures the equipment and the working area of the worker;
a step (b) of recognizing an operating state of the equipment and a working state of the worker from the image of the image data and recognizing an article in the image of the image data;
The operation state of the equipment is collected as equipment information, the work state of the worker as worker information, and the type and state of an article as article information, and the photographing is performed based on the facility information, the worker information, and the article information. (c) determining the operating procedure for the process within the scope ;
A process information acquisition method.

(7) The step (a) recognizes the operating state of the equipment from a change in color or a change in the light emitting position of the signal lamp shown in the image of the image data, and recognizes the operation state shown in the image of the image data. The process information acquisition method according to (6) above, wherein the work state of the worker is recognized from the position and/or orientation of the worker.

(8) the camera has a wide-angle lens or a fisheye lens;
(6) above, comprising a step (d) of correcting any distortion in the image taken using the wide-angle lens or the fisheye lens between the step (a) and the step (b); Or the process information acquisition method as described in (7).

(9) The process information acquisition method according to any one of (6) to (8) above, further comprising a step (e) of storing the image data for a predetermined period in a storage unit.

(10) Furthermore, the step (f) of calculating the resource used for producing the product by the equipment and/or the lead time required for production from the equipment information, the worker information, and the article information collected. The process information acquisition method according to any one of (6) to (9) above.

(11) A process information acquisition program for causing a computer to execute the process information acquisition method according to any one of (6) to (10) above.

According to the present invention, by recognizing the operating state of the equipment, the working state of the worker, and the article from the image captured by the camera, the use of multiple sensors and electronic tags carried by the worker is eliminated. . As a result, the initial investment can be suppressed, there is no burden on workers, and information on equipment, workers, and goods can be obtained without the need for time-series matching of data.

BRIEF DESCRIPTION OF THE DRAWINGS It is explanatory drawing for demonstrating the structure of the process information acquisition system of embodiment. It is a block diagram for explaining functional composition of a process information acquisition system of an embodiment. FIG. 4 is an explanatory diagram for explaining detection of a worker's position and orientation using skeleton recognition technology; 4 is a flow chart showing a processing procedure;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and overlapping descriptions are omitted. Also, the dimensional ratios in the drawings are exaggerated for convenience of explanation, and may differ from the actual ratios.

FIG. 1 is an explanatory diagram for explaining the configuration of the process information acquisition system of the embodiment. FIG. 2 is a block diagram for explaining the functional configuration of the process information acquisition system of the embodiment. It should be noted that the equipment, work table, and size, orientation, arrangement, etc. of the workers depicted in FIG. It does not have a simple structure.

A process information acquisition system 1 has a server 10 and a camera 20 connected to the server 10 .

The camera 20 includes the signal lamp 51 of the equipment 50 and the working range of the worker as a photographing range. In this embodiment, the photographing range of the camera 20 includes, in addition to the signal lamp 51 of the facility 50, the operation panel 52, the door 53, and the work table 55 of the facility 50. FIG. The work range of the worker includes these and is the range necessary for the worker to work on them.

Equipment 50 may be any equipment, for example, an automatic processing machine such as a numerical control processing machine. Also, in the following description, production includes processing of articles by an automatic processing machine.

The camera 20 is installed at a position for photographing the range in which the worker works, including the equipment 50 and the work table 55, from above, for example. By photographing the worker from above, the direction in which the worker extends his arm is recognized in the photographed image. Therefore, in this embodiment, the orientation of the worker is recognized from the direction in which the worker sticks out his or her arm (details will be described later). Furthermore, it is more preferable that the camera 20 is installed at a position where the operation panel 52 of the equipment 50 can be photographed. In particular, camera 20 is more preferably installed at a position where the display on operation panel 52 can be photographed. For these reasons, in this embodiment, the camera 20 is installed obliquely above the equipment 50 so that the operator can be photographed from above, and the operation panel 52 can also be photographed. From the contents displayed on the operation panel 52, in addition to the information of the signal lamp 51, the state of the equipment 50 can be recognized in more detail.

The camera 20 is not particularly limited as long as it can capture moving images or continuous still images. Specifically, as the camera 20, for example, a general movie camera or still camera can be used. The camera 20 has an interface (not shown) for communicating with the server 10, as will be described later. Image data is output from the camera 20 and transmitted to the server 10 . The image data is data representing a moving image or a series of still images. In the present embodiment, moving images are taken. In the case of shooting continuous still images, the shooting interval should be set to a time interval at which the work state can be recognized. If this time interval is too long, it becomes difficult to recognize the work state. Although it depends on the content of the work, it is preferable to set the photographing interval to 10 seconds or less, for example.

Also, the camera 20 preferably has, for example, a wide-angle lens or a fish-eye lens so that a single camera 20 can photograph the entire working range. If you have a wide-angle lens or a fisheye lens, the captured image may be distorted. If a wide-angle lens or a fish-eye lens is provided, it is preferable to provide an image correction unit that corrects image distortion specific to the wide-angle lens or fish-eye lens, either on the camera 20 side or on the server 10 side, or separately from these. When a wide-angle lens or a fish-eye lens is provided, image processing can be performed efficiently by using image data subjected to image correction by the image correction unit. In this embodiment, the server 10 has the function of the image corrector 115 (see FIG. 2). The function of the image correction unit 115 can be realized by software.

The server 10 has, as shown in FIG. 2, a computing unit 11, a storage unit 12, and an interface 13 (I/F).

The computing unit 11 has, as its functions, an equipment state recognition unit 111 that recognizes the operating state of the equipment 50 from the image data, a work recognition unit 112 that recognizes the work state of the worker, an article recognition unit that recognizes the type and state of the article. 113, an analysis processing unit 114 that analyzes the operating state of the entire process from the operating state of the equipment 50, the work state of the worker, and the type and state of the article. The details of the function of each part will be described later. Moreover, the calculation unit 11 has an image correction unit 115 as necessary. The functions of the image correction unit 115 are as described above.

The calculation unit 11 is a processor (CPU: Central Processing Unit or GPU: Graphics Processing Unit) that executes various types of calculation processing in the server 10 .

The storage unit 12 is a storage device provided in the server 10, and is configured by appropriately combining, for example, ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), and the like. ROM is a read-only storage device that stores various programs and various data. RAM is a high speed random access storage device that temporarily stores programs and data as a working area. The HDD is a large-capacity random access storage device that stores various programs and data. Of course, the storage unit 12 may be another storage device.

Interface 13 receives input of image data from camera 20 . The interface 13 is, for example, a wired communication interface such as Ethernet (registered trademark), SATA, PCI Express, and USB, Bluetooth (registered trademark) according to standards such as IEEE1394, IEEE802. 11, and the like. The interface 13 of the server 10 is used not only to receive input from the camera 20, but also to transmit and receive data to and from other external devices. Image data input to the server 10 is described as digital data in this embodiment, but is not limited to this. If the image data output by the camera 20 is analog data, a device for converting it into digital data may be provided between the camera 20 and the server 10 or between the camera 20 and the server 10 . If the server 10 receives analog data, the server 10 has an interface 13 for analog lines.

The server 10 is a general computer, and each function is achieved by executing a program for the computer serving as the server 10 to implement the function of each part.

The functions of the server 10 are further described below.

The calculation unit 11 causes the storage unit 12 to store the received image data. The storage period is selected as required, for example, from several hours to several days or several weeks. Needless to say, the storage unit 12 has a capacity necessary for storing the image data.

The facility state recognition unit 111 recognizes the operating state of the facility 50 by recognizing the color of the signal lamp 51 provided in the facility 50 from the image indicated by the image data.

The equipment state recognizing unit 111 reads the image data almost at the same time as the image data is stored in the storage unit 12, and processes it in real time. Real-time processing determines the current operational status.

The signal light 51 is also called a laminated signal light, a signal tower, or the like, and has, for example, green, yellow, and red lamps that emit light in accordance with the operating state of the equipment 50 . Each color of the signal lamp 51 indicates, for example, the following states. Green indicates that the facility 50 is operating normally. Yellow indicates a normal pause status. Red indicates a state of abnormal stop due to the occurrence of a problem or the like. Further, when the signal lamp 51 is turned off, it indicates that the facility 50 is normally stopped. In the work process, when an article to be processed is placed on or removed from the facility 50, the power supply to the facility 50 is normally turned off for safety. In such a case, the signal lamp 51 is extinguished. It should be noted that depending on the facility 50, there is a facility in which an article can be placed or taken out without turning off the power. In the case of such equipment 50, the signal lamp 51 lights up or flashes in yellow when an article is placed or taken out. Also, when the facility 50 is stopped systematically, the signal lamp 51 is extinguished. The emission color of the signal lamp 51 and the state of the facility 50 at that time can be arbitrarily set, and are not limited to the above combinations.

The equipment state recognition unit 111 sets in advance a predetermined area in which the signal lamp 51 is shown in the image, and recognizes the operating state of the equipment 50 from the color change in the predetermined area. Note that the equipment state recognition unit 111 may recognize the operating state of the equipment 50 from the light emitting position instead of the color of the signal lamp 51 . The signal lamps 51 are, for example, arranged in order of red, yellow, and green from the top.

Also, the equipment state recognition unit 111 can recognize the operating state of the equipment 50 from the display on the operation panel 52 . In this case, the equipment state recognition unit 111 preliminarily sets an area in which the operation panel 52 is shown in the image, and recognizes characters in that area. In this embodiment, both the camera 20 and the facility 50 are fixed objects, so even if the image is a moving image, the area where the operation panel 52 is captured does not move. For example, if the facility 50 is an automatic processing machine, the characters displayed on the operation panel 52 are not so diverse. The characters displayed on the operation panel 52 are, for example, time, code numbers indicating the current state, and the like. These are often alphanumeric abbreviations or numerical values. Therefore, the equipment state recognition unit 111 only needs to be able to recognize alphanumeric characters. Of course, the equipment state recognition unit 111 may recognize not only alphanumeric characters but also hiragana, katakana, and kanji characters from images.

For example, when the signal lamp 51 is green and a number is displayed on the operation panel 52, the number is recognized as the time until the end of the work or the elapsed time from the start of the work. Either the time until the end of work or the elapsed time can be set in advance, or if the value decreases with the passage of time, it can be determined as the time until the end of work, and if it increases, it can be determined as the elapsed time. good. Also, when the signal lamp 51 is yellow or red and a number is displayed on the operation panel 52, it is recognized that a code indicating the reason for the stop is displayed. For example, from the code displayed when the signal light 51 is yellow, it may be recognized that the facility 50 is stopped for confirmation during processing, or stopped due to the end of processing. Also, when the signal lamp 51 is red, it is recognized that a code indicating an abnormality is being displayed. The storage unit 12 stores the numerical values of these codes and their corresponding reasons. The equipment state recognition unit 111 compares the read code with the reason stored in the storage unit 12 to determine the reason for stopping. The determined reason for stopping is transmitted to, for example, a computer for process control (not shown), so that the process manager can be informed of the stop of the facility 50 and the reason.

Further, for example, in an automatic machine tool, the operation panel 52 displays voltage values, current values, and other numerical values. In such equipment 50, by recognizing the characters (numerical values) displayed on the voltage and current values, the state of the machining that is currently being executed can be recognized in detail as the operating state of the equipment 50. FIG.

As another mode, for example, an image showing the operation panel 52 may simply be sent to a computer for process control or the like. In this case, characters may not be recognized from the display on the operation panel 52 . Also, in this case, the area of the operation panel 52 may be set in advance so that it is enlarged and displayed on the computer for process control.

Furthermore, by monitoring the operation panel 52 , the equipment state recognition unit 111 can also recognize input operations on the operation panel 52 .

As described above, in this embodiment, by reading not only the color of the signal lamp 51 but also the display of the operation panel 52, it is possible to recognize the operating state of the equipment 50 in more detail.

The equipment state recognition unit 111 can also recognize the operating state of the equipment 50 by recognizing the opening/closing state of the door 53 from the position of the door 53 of the equipment 50 . The open/closed state of the door 53 is recognized by, for example, extracting the feature of the door 53 from the image and determining that the position of the feature is different between the frames of the images arranged in time series. The feature of the door 53 is a feature amount for identifying the door 53 from the image. Such a feature amount may be, for example, the door 53 as a whole, or an accessory that moves with the door 53 (for example, a handle). The feature amount is stored in advance in the storage unit 12 and used when identifying the door 53 from the image.

Specifically, for example, if the door 53 is open, it is recognized that an article has been taken in and out of the equipment 50 . As for the combination with the signal lamp 51, for example, if the light is turned off before the door 53 is opened, it is recognized that the article has been placed in the facility 50 when the door 53 is opened. Also, the light is green until the door 53 is opened, and after that, the light is turned off. Also, when the door 53 is opened when the color is yellow, it is recognized that the article being processed has been confirmed.

Thus, in this embodiment, by further recognizing the open/closed state of the door 53, it is possible to recognize the operating state of the facility 50 in more detail.

In this embodiment, the information of the facility 50 obtained by the facility state recognition unit 111, that is, the light emission color (including extinguished) of the signal lamp 51, the character string and operation recognized from the operation panel 52, and the open/closed state of the door 53. It is collectively called equipment information.

Next, the work recognition unit 112 detects the position and orientation of the worker from the image data, and recognizes whether the work is for the equipment 50 (equipment work) or other work (non-equipment work).

An existing technique can be used to detect the position and orientation of the worker from the image data. Existing technologies include, for example, Open Pose (see https://github.com/CMU-Perceptual-Computing-Lab/openpose) and Deep Pose (https://www.slideshare .net/mitmul/deeppose-human-pose-estimation-via-deep-neural-networks etc.)) can be used. Open pose and deep pose are techniques for recognizing postures by estimating a human skeleton from two-dimensional image data. Such techniques are called skeleton recognition techniques. In addition, a technique using a depth camera (RBG-D camera) or TOF (Time of Flight) can be used. In this case, as the camera 20, a dedicated camera capable of image recognition including distance may be used instead of a normal movie camera.

A specific example will be given to describe the detection of the position and orientation of a worker using the skeleton recognition technology. FIG. 3 is an explanatory diagram for explaining detection of a worker's position and orientation using the skeleton recognition technology.

The photographing range in FIG. 3 is set so as to overlap with the area where the worker works. Therefore, the operator works within the photographing range. A range in which a worker works is called a work range. Alternatively, the photographing range may be set wider than the work range, and the work recognition unit 112 may detect only workers within the work range preset in the image of the image data. .

First, the work recognition unit 112 detects a worker within the imaging range. After detecting the worker, the work recognition unit 112 acquires the coordinate values of the skeleton of the main parts such as the head and shoulders of the worker. The position of the worker is detected from the coordinate values of the main skeleton. In the skeleton recognition technology, a coordinate system necessary for processing is set in advance within the imaging range. For example, a two-dimensional coordinate system is set, but a three-dimensional coordinate system may also be set. Also, the position of the worker may be determined from the coordinate values of the skeleton of the legs, for example, in addition to the head and shoulders.

Next, the work recognition unit 112 determines the direction in which the arm (especially the portion from the elbow) protrudes with respect to the skeleton of the main parts such as the head and shoulders, and determines the direction in which the worker is facing. judge there is. In this embodiment, the orientation of the operator is determined from the orientation of at least one arm. While working on the equipment 50 or the like, the operator may have both arms protruding toward the equipment 50, or may work with one hand. One-handed work specifically includes, for example, changing program settings and restarting. Such work can be performed with one hand on the operation panel 52 of the equipment 50 . At this time, the worker lowers the other arm along his/her body. In such a case, one arm would project in the direction of installation 50 while the other arm would not. Therefore, in this embodiment, as described above, the orientation of the worker is determined from the direction in which at least one of the arms is directed, and the working state of the worker is determined therefrom.

In FIG. 3, for example, the direction in which the worker 30a projects one arm is the direction of the facility 50, so it can be determined that the worker 30a is facing the facility 50. FIG. Therefore, the work recognition unit 112 determines that the worker 30a is working on the equipment 50 .

In addition, since the direction in which both arms of the worker 30b protrude is the direction of the work table 55, it can be determined that the worker 30b faces the work table 55. FIG. Therefore, the work recognition unit 112 determines that the worker 30 b is working on the work table 55 .

Furthermore, the direction in which both arms of the worker 30 c protrude is neither the direction of the equipment 50 nor the direction of the work table 55 . In such a case, the work recognition unit 112 determines that the worker 30c is not working within the imaging range, that is, within the work range.

In this embodiment, the recognition result of the worker's position and orientation obtained by the work recognition unit 112 is referred to as worker information.

Next, the item recognition unit 113 recognizes the type and state of the item from the image. Article recognition is performed, for example, by image processing. The article recognition unit 113 recognizes the shape of an object in the image by image analysis, and compares it with a predetermined article feature (feature amount) stored in advance. As a result of the comparison, if the feature difference is within a predetermined range, the article recognition unit 113 recognizes the object recognized in the image as the article of the same type as the predetermined article. In the object recognition, it takes a long time to process all the objects in the image. Therefore, in the present embodiment, the article is recognized in a predetermined area within the image. The predetermined area is, for example, above the work table 55 and near the door 53 of the equipment 50 . As a result, the article recognition unit 113 recognizes the presence or absence of an article on the work table 55 as the state of the article, and if there is an article, the type of the article is recognized. Also, the predetermined area may be the vicinity of the wire through which the worker moves from the work table 55 to the equipment 50 . By determining a predetermined area for recognizing an article in this way, the recognition speed can be increased. In addition, the feature (feature amount) stored as a predetermined article is not limited to one or one kind, but by storing the features (feature amounts) of various articles of different types, various articles of different types can be stored. can be recognized.

As another method for recognizing an article, for example, the article recognition unit 113 is caused to read a mark, an article code, a bar code, or the like drawn on a sticker attached to an article or a tag attached to the article. may Barcodes may be one-dimensional or two-dimensional. It is preferable to determine the range in which the seal or tag is read from the image in advance. The range in which the sticker or tag is read is, for example, the work table 55, the vicinity of the door 53 of the equipment 50, the vicinity of the conductor of the operator, and the like.

In this embodiment, the product recognition result obtained by the product recognition unit 113 is referred to as product information.

Next, the analysis processing unit 114 calculates the shooting range based on the equipment information collected by the equipment state recognition unit 111, the worker information collected by the work recognition unit 112, and the article information collected by the article recognition unit 113. Obtain the operating status of the entire process within. Also, the analysis processing unit 114 performs various analyzes based on this information.

A predetermined determination table, for example, is used to obtain the operating state of the entire process. The determination table is table data describing combinations of facility information, worker information, and article information, and work procedures corresponding to the combinations.

Table 1 is an example of a determination table.

Description will be made with reference to Table 1.

For example, in procedure (1), the content of worker information is worker recognition, the content of equipment information is that signal light 51 is turned off, the operation panel 52 and door 53 are not recognized, and the content of article information is The article is recognized on the work table 55 . At this stage, since the worker is recognized for the first time, the content of the worker information is the worker recognition. At this time, the number of workers is also specified. Also, the facility 50 is stopped, and the article is recognized on the work table 55 for the first time. Accordingly, it is determined that selection of articles to be processed from now on is being performed.

Subsequently, in the procedure (2), the content of the worker information is equipment work, the content of the equipment information is that the signal light 51 is turned off, the operation panel 52 is not recognized, and the door 53 is recognized that the door is open/closed. The content of the article information is not recognized. At this stage, there are no articles left on the work table 55, the door 53 of the facility 50 is opened and closed, and the worker is working. Moreover, the equipment 50 is in the process of stopping. Then the procedure is after the selection of the article. From these states, it is determined that an article has been installed in the facility 50 . Accordingly, it is determined that the article is being placed. The equipment work is, for example, determined that the worker is working on the equipment 50 based on the information that the worker is heading to the equipment 50 .

Subsequently, in the procedure (3), the content of the worker information is equipment work, the content of the equipment information is that the signal light 51 is turned off, the operation panel 52 recognizes the input operation, and the door 53 does not recognize it. The content of the article information is not recognized. At this stage, it is determined that the operator is operating the operation panel 52 after installing the article in the facility 50 . Moreover, the equipment 50 is in the process of stopping. Accordingly, it is determined that the equipment is being operated.

Next, in procedure (4), the content of the worker information is equipment work, the content of the equipment information is green for the signal light 51, the non-recognition for the operation panel 52 and door 53, and the content of the item information is There is no recognition of goods. At this stage, it is determined that the facility 50 has started operating normally. As a result, it is determined that the facility is in operation.

Subsequently, in procedure (5), the content of the worker information is equipment work or table work (equipment work/table work), the content of equipment information is that the signal light 51 is green, and the operation panel 52 and door 53 are not recognized. , and the content of the product information is that the product is not recognized. At this stage, since the equipment 50 is operating normally, it is determined that the worker is doing other work. As a result, the work is determined to be incidental work. The other work is various, for example, measuring other articles, checking the operating state of the equipment 50, and the like.

Subsequently, in procedure (6), the content of the worker information is equipment work, the content of the equipment information is that the signal light 51 is yellow, the operation panel 52 and the door 53 are not recognized, and the content of the product information is There is no recognition of goods. At this stage, it is determined that the equipment 50 is normally stopped and the operator is looking inside the equipment 50 to confirm the processing state of the article. Accordingly, it is determined that the article has been confirmed.

Subsequently, in the procedure (7), the content of the worker information is equipment work, the content of the equipment information is that the signal lamp 51 is yellow, the operation panel 52 recognizes the input operation, and the door 53 does not recognize, The content of the product information is no recognition of the product. At this stage, it is determined that preparations are being made to continue the operation of the facility 50 . Accordingly, it is determined that the equipment is being operated.

Next, in procedure (8), the content of the worker information is equipment work, the content of the equipment information is green for the signal light 51, the non-recognition for the operation panel 52 and the door 53, and the content of the item information is There is no recognition of goods. At this stage, it is determined that the facility 50 is operating normally. As a result, it is determined that the facility is in operation.

Here, procedures (6) to (8) may be repeated.

Subsequently, in procedure (9), the content of the worker information is equipment work, the content of the equipment information is that the signal light 51 is turned off, the operation panel 52 is not recognized, and the door 53 is recognized that the door is opened. The content of the product information is no recognition of the product. At this stage, it is determined that the facility 50 is normally stopped and the door 53 is being opened and closed. Moreover, it is determined that the article is being taken out because the processing has been completed and stopped from the flow of the procedure. As a result, it is determined that the article has been removed.

Subsequently, in procedure (10), the content of worker information is table work, the content of equipment information is that the signal lamp 51 is turned off, the operation panel 52 and door 53 are not recognized, and the content of article information is The article is recognized on the work table 55 . At this stage, it is determined that the worker is working on the work table 55 . Moreover, it is determined that the article is being measured because the processing has been completed from the flow of the procedure. As a result, it is determined that the article is measured.

Subsequently, in procedure (11), the content of the worker information is equipment work, the content of the equipment information is that the signal lamp 51 is turned off, the operation panel 52 and the door 53 are not recognized, and the content of the product information is It is not recognized. At this stage, workers are working on equipment. Moreover, it is determined that the article is being washed in the equipment 50 because the measurement of the article has been completed from the flow of the procedure. As a result, it is determined that the articles are washed.

Subsequently, in procedure (12), the content of worker information is table work, the content of equipment information is that the signal lamp 51 is turned off, the operation panel 52 and door 53 are not recognized, and the content of article information is The article is recognized on the work table 55 . At this stage, it is determined that the worker is working on the work table 55 . Moreover, it is determined that the article is being inspected here because it is after article washing has ended from the flow of the procedure. As a result, it is determined to be an article inspection.

In addition, the determination table also describes out-of-procedure determinations and their combinations.

For example, in non-procedure 1, the content of the worker information is equipment work, the content of the equipment information is that the signal light 51 is turned off, the operation panel 52 is not recognized, the door 53 is recognized as opening and closing the door, and the product information The contents are not recognized. When such a combination of information is recognized, if the signal light 51 was red in the stage prior to out-of-procedure 1, it is determined that the repair is in progress.

In addition, in the out-of-procedure 2, the contents of the worker information are not recognized, and as the contents of the equipment information, the signal lamp 51 is red, the operation panel 52 is recognized as an error display, the door 53 is not recognized, and the item information is not recognized. The contents are not recognized. If such a combination of information is recognized, there is a possibility that nothing has been done even though the facility 50 has stopped for some reason. If such a combination of information is recognized, it is determined that work is delayed.

It should be noted that the contents of the determination table described above are merely examples for explanation, and of course, they change depending on various processes and processing details of articles.

Next, an example of information analysis by the analysis processing unit 114 will be described.

For example, the analysis processing unit 114 aggregates each piece of information into one dashboard. The analysis processor 114 also calculates the resources expended to produce the product. Calculated resources are used, for example, for cost accounting. Also, the analysis processing unit 114 calculates the lead time required for producing the product and uses it as an actual value. The calculated actual value is used, for example, for budget management that compares the target budget and scheduled time with actual results, and for improving the accuracy of estimates when acquiring projects.

As a specific analysis, for example, the resource is calculated as the number of workers spent per piece of equipment from the number of workers obtained in the selection of items in the work procedure (1) described above. Also, the resource may be the total number of workers who have spent on work procedures (1) to (12). If the voltage value or current value is read from the operation panel 52 , the resource may be used as the resource of the facility 50 by obtaining the power amount from those values.

As the production lead time, the total production time can be obtained from the time required for the work procedures (1) to (12) described above. Also, the actual time during which the facility 50 is being used can be obtained from the time taken for the work procedures (4) to (8).

In this embodiment, a plurality of pieces of information such as facility information, worker information, and article information are collected from one piece of image data. Therefore, each piece of information is originally collected from one piece of image data, and therefore has the same time series. Therefore, when each piece of information is aggregated into one dashboard, chronological alignment is not required. A plurality of aggregated pieces of information are transmitted to, for example, a computer for process control. The computer for process control uses the received information for further analysis.

Next, a processing procedure for acquiring and analyzing each piece of information will be described. FIG. 4 is a flow chart showing a processing procedure. In this processing procedure, the computing unit 11 (specifically, the CPU) executes a program created based on this processing procedure, so that each unit operates as a function of the computing unit 11 described above.

First, the calculation unit 11 receives and acquires image data from the camera 20 (S1). The acquired image data is stored in the storage unit 12 .

Subsequently, the calculation unit 11 reads the stored image data and collects equipment information, worker information, and article information from the image (S2).

Subsequently, the computing unit 11 compiles the collected information into one for easy analysis, and stores this in the storage unit 12 (S3). At this stage of S3, if there is information that has already been aggregated, the calculation unit 11 updates that information.

After S3, if there is no input to end the processing (S4: NO), the calculation unit 11 returns to S1 as it is, continues to receive image data sequentially, and continues each step. If the end of processing is input (S4: YES), the computing unit 11 ends the processing.

According to this embodiment, the following effects are obtained.

In this embodiment, from image data obtained by one camera 20, a plurality of pieces of information such as facility information, worker information, and article information of the facility 50 are obtained. Therefore, in the present embodiment, a plurality of pieces of information necessary for process control are originally obtained from one piece of image data, and thus are in the same time series. No need to line up the series. Therefore, in this embodiment, information can be processed more easily than in the prior art, and the processing speed can be increased.

Also, in this embodiment, from image data obtained by one camera 20, a plurality of pieces of information such as facility information, worker information, and article information of the facility 50 are obtained. For this reason, unlike the prior art, the present embodiment does not provide a sensor for detecting the operating state of the facility 50 or use an electronic tag carried by the worker. Information can be obtained. Therefore, in this embodiment, it is only necessary to introduce the camera 20, and the existing server 10 can be used for image data processing. Alternatively, even if the server 10 is newly introduced, it is sufficient to introduce only a general server computer. Therefore, unlike the conventional technology, this embodiment does not require a separate host system for collecting information from a plurality of sensors or electronic tags. Therefore, this embodiment requires less initial investment than the prior art. As a result, this embodiment can collect a plurality of pieces of information in the process while suppressing the initial investment, and can be used for various analyzes such as process management and budget/actual management, and can be used for appropriate improvement actions.

Also, in this embodiment, data for obtaining a plurality of pieces of information is one piece of image data, so there is no need to store a plurality of pieces of information as in the prior art. Therefore, this embodiment can reduce the amount of data. Moreover, in the present embodiment, since only one image data is transmitted and received, it is possible to reduce the load on the band associated with data transmission and reception.

Although the embodiments of the present invention have been described above, the present invention can be modified in various ways.

In the embodiment, an example in which the server 10 is used for collecting and analyzing information has been described. Therefore, this small computer may execute a program for achieving each function described above. By doing so, the present invention can be implemented only by installing the camera 20 .

Further, each function of the server 10 described above may be replaced by a cloud server. For this, for example, the camera 20 is provided with an interface 13 that can be connected to the Internet, and the image data is directly transmitted to the cloud server. By executing the already explained processing on the cloud server, each information can be analyzed on the cloud server, and each information and analysis results can be browsed from the terminal of the user at a remote location.

Further, in the embodiment, the imaging range of the camera 20 is an area including one facility 50, but the imaging range may be such that a plurality of facilities 50 can be photographed at once. In this case, from the orientation of the worker facing each facility 50, it is determined which facility 50 the worker is working on. Alternatively, a work range may be provided for each piece of equipment 50 and it may be determined that a worker within each work range is working on the piece of equipment 50 .

In the embodiment, the working state of the worker is determined from the position and orientation of the worker. You may judge that you are doing equipment work. Conversely, it may be judged only from the orientation of the operator. In particular, when the imaging range is narrow, for example, when the range of movement of the worker is about 5 steps or less, the worker performs equipment work when the worker faces the equipment 50, and performs other work when the worker does not face the equipment 50. Or you can judge that you are not working.

Further, in the embodiment, the three-color signal lamp 51 is used. For example, for a two-color traffic light, green for normal operation, yellow for normal stop or red for abnormal stop, and so on. When adapting the embodiment to such equipment 50, the process for yellow or red should be avoided. Furthermore, in the case of 1 color, 4 colors, 5 colors, etc., it is sufficient to process according to the operating state of the equipment 50 .

Further, in the embodiment, input operations to the operation panel 52 and opening/closing of the door 53 are recognized by the equipment state recognition unit 111 described as functions of the calculation unit 11 . However, such a function is merely an example. An input operation to the operation panel 52 or opening/closing of the door 53 may be recognized by the operator's action such as entering.

In the embodiment, as equipment information, not only the signal lamp 51 but also the status of the operation panel 52 and the door 53 are recognized.

In addition, the conditions and numerical values used in the description of the embodiments are for the purpose of explanation only, and the present invention is not limited to these conditions and numerical values.

Further, the computer for process control mentioned in the embodiment is a computer connected to the server 10, for example, a computer provided at a work site or a computer provided in a remote process control department. The computer for process control may be, for example, a tablet owned by the process manager or worker, or a portable terminal such as a smart phone.

Moreover, the program according to the present invention can also be realized by a dedicated hardware circuit. In addition, the program may be provided by a computer-readable recording medium such as a USB (Universal Serial Bus) memory or a DVD (Digital Versatile Disc)-ROM (Read Only Memory), or may be provided by a network such as the Internet regardless of the recording medium. It is also possible to provide online through In this case, the program is usually stored in a magnetic disk device or the like that constitutes the storage unit. Also, the program can be provided as independent application software, or can be provided by being incorporated into another software as one function.

Furthermore, the present invention can be variously modified based on the configuration described in the claims, and these are also within the scope of the present invention.

1 process information acquisition system,
10 servers,
11 calculation unit,
12 storage unit,
13 interface,
20 cameras,
50 equipment,
51 signal lights,
52 operation panel,
53 door,
55 working table,
111 equipment state recognition unit,
112 work recognition unit,
113 article recognition unit;
114 analysis processing unit,
115 image corrector;

Claims (11)

a camera that includes the equipment and the working range of the worker in the shooting range;
Acquiring image data from the camera, recognizing the operating state of the equipment and the working state of the worker from the image of the image data, and recognizing an article within a shooting range from the image of the image data , the operation status of the facility is collected as equipment information, the work status of the worker as worker information, and the type and state of the article as article information, and based on the facility information, the worker information, and the article information, a calculation unit that determines a work procedure for a process within the imaging range ;
A process information acquisition system. The computing unit recognizes the operating state of the equipment from a change in color or a change in the light emitting position of the signal lamp shown in the image of the image data, and the position and/or position of the worker shown in the image of the image data. 2. The process information acquisition system according to claim 1, wherein the work state of the worker is recognized from orientation. the camera comprises a wide-angle lens or a fisheye lens;
3. The process information acquisition system according to claim 1, further comprising an image corrector for correcting distortion of an image captured using a wide-angle lens or a fisheye lens. a storage unit that stores the image data;
4. The process information acquisition system according to any one of claims 1 to 3, wherein said calculation unit stores said image data for a predetermined period in said storage unit. 1-, wherein the calculation unit calculates the resources used to produce the product by the equipment and/or the lead time required for production from the equipment information, the worker information, and the article information collected. 5. The process information acquisition system according to any one of 4. (a) acquiring image data from a camera that captures the work area of the equipment and workers;
a step (b) of recognizing an operating state of the equipment and a working state of the worker from the image of the image data and recognizing an article in the image of the image data;
The operating state of the equipment is collected as equipment information, the work state of the worker as worker information, and the type and state of an article as article information, and the photographing is performed based on the facility information, the worker information, and the article information. (c) determining the operating procedure for the process within the scope ;
A process information acquisition method. The step (a) recognizes the operating state of the equipment from a change in color or a change in the light emitting position of the signal lamp shown in the image of the image data, and the position of the worker shown in the image of the image data. 7. The process information acquisition method according to claim 6, wherein the work state of the worker is recognized from and/or orientation. the camera comprises a wide-angle lens or a fisheye lens;
7. The method according to claim 6, further comprising a step (d) of correcting distortion, if any, in an image taken using said wide-angle lens or said fisheye lens, between said step (a) and said step (b). 7. The process information acquisition method according to 7. 9. The process information acquiring method according to claim 6, further comprising a step (e) of storing said image data for a predetermined period in a storage unit. Furthermore, a step (f) of calculating the resource used for producing the product by the equipment and/or the lead time required for production from the equipment information, the worker information, and the article information collected. Item 10. The process information acquisition method according to any one of items 6 to 9. A process information acquisition program for causing a computer to execute the process information acquisition method according to any one of claims 6 to 10.

Images