WO2020246082A1 - Work monitoring device and work monitoring method - Google Patents

Abstract

The present invention monitors work conducted by a person with high accuracy by an object recognition technique without preparing large-volume data in advance. A work monitoring device acquires image data of a state in which a person conducts work, acquires positions in the image data of each of a plurality of objects in the image data, generates a determination result table including information which indicates a result of determination of whether one or more determination subject areas as areas set for the image data respectively include the positions of the plurality of objects or not, stores determination result pattern/work content correspondence information as information acquired by associating the determination result table generated on the basis of the image data and work content information as information which indicates the work conducted by the person in the image data, generates a determination result table for image data newly acquired, and specifies work conducted by the person by comparing the generated determination result table and the determination result pattern/work content correspondence information, thereby monitoring the work conducted by the person.

Description

Work monitoring device and work monitoring method

The present invention relates to a work monitoring device and a work monitoring method.

This application claims priority based on Japanese Patent Application No. 2019-103551 filed on June 3, 2019, and incorporates the entire disclosure into this application.

Patent Document 1 describes a method of recognizing the motion of a moving body for the robot to recognize the posture, motion, or behavior of a person. In the above motion recognition method, the eigenspace data A in which the frame image data A for each basic motion of the moving object A is displayed by dots is created in advance and created in a database, and the frame image data B of the target moving object B is displayed by dots. The eigenspace data B is compared with the eigenspace data A for each basic operation, the eigenspace data A closest to the eigenspace data B is selected, and the operation of the moving body B is recognized. For each frame image data A for each basic motion, the moving body A is made to perform the basic motion, and the moving body A performing the basic motion is photographed from multiple directions using a plurality of image input means, and is continuously acquired for each image input means. It is obtained from a compressed image created by weighting each of a plurality of frame images and then superimposing them.

Patent Document 2 describes a motion recognition system that recognizes the shape and motion of a hand in a user interface that uses a person's gesture or gesture. The motion recognition system recognizes the shape and motion of the target by processing the time-series image data including the image of the specific target, extracts the moving region from the time-series image data, and performs the time-series image data. A region containing a color that characterizes the target is extracted from, and a region that is a moving region and that includes a color that characterizes the target is extracted as the target region.

Non-Patent Document 1 describes various behavior recognition technologies that utilize sensors and cameras. Non-Patent Document 2 describes that motion recognition is performed using "Zero Shot Learning" that predicts labels of data that do not appear in the training data.

JP-A-2009-176059 Japanese Unexamined Patent Publication No. 2001-16606

At the product manufacturing site, the introduction of a mechanism to monitor whether or not the worker is working according to the correct work procedure by recognizing the movement of the worker for the purpose of quality control and ensuring the safety of the worker is being promoted. ing. In recent years, technology using machine learning has been attracting attention, and for example, research and development of motion recognition technology using a database (“UCF101-Action Recognition Data Set”, etc.) that is open to the public is being actively carried out.

By the way, since the work performed by the worker differs from site to site, when introducing a work monitoring mechanism using motion recognition technology to the site, it is necessary to collect data on various tasks performed by the worker at each site. .. Even when using the above database, it is necessary to collect data for each site for unregistered work.

For example, the technique described in Patent Document 1 generates eigenspace data in advance with a frame image for each basic operation, compares the distance from the characteristics of the operation of the eigenspace, and performs motion recognition. It is necessary to prepare a large amount of data in advance for the basic operation, which puts a heavy load on the workers at the site at the time of introduction.

On the other hand, in the technique described in Patent Document 2, motion recognition is performed by extracting a target region based on a motion region and a region containing a color that characterizes the motion recognition target using time-series image data and analyzing the shape. It is a method that does not require data in advance. However, since the same technology recognizes the movement based on the shape, it is not possible to accurately recognize the position of the working hand and the surrounding information.

In addition, the technique described in Non-Patent Document 2 predicts labels of data that do not appear in the learning data, but it is necessary to prepare the data necessary for predicting the recognition target at the site.

The present invention has been made based on such a background, and provides a work monitoring device and a work monitoring method capable of accurately monitoring the work performed by an operator without preparing a large amount of data in advance. The purpose is.

One of the present inventions for achieving the above object is a work monitoring device, which is configured by using an information processing device having a processor and a memory, and displays image data showing a state in which an operator is performing work. An image data acquisition unit to be acquired, an object position detection unit that acquires information indicating the position of each of a plurality of objects reflected in the image data in the image data, and one area set in the image data. A determination result table generation unit that generates a determination result table including information indicating the result of determining whether or not the positions of the plurality of objects are included in each of the above determination target areas, and the image data. A storage unit that stores the judgment result pattern / work content correspondence information, which is information corresponding to the judgment result table generated based on the above and the work content information which is information indicating the work in the image data, and a newly acquired storage unit. A monitoring processing unit that generates the determination result table for the image data and compares the generated determination result table with the determination result pattern / work content correspondence information to specify the work performed by the worker. , Equipped with.

In addition, the problems disclosed in the present application and the solutions thereof will be clarified by the column of the form for carrying out the invention and the drawings.

According to the present invention, it is possible to accurately monitor the work performed by the worker without preparing a large amount of data in advance.

It is a figure which shows the schematic structure of the work monitoring system. It is a figure which shows the hardware configuration example of the work monitoring apparatus. It is a figure which shows the main function which a work monitoring apparatus has. (A) is an example of image data, (b) is a diagram drawn by extracting the position of a marker reflected in the image data and a judgment target area, and (c) is a judgment generated based on the image data. It is a result table. (A) to (c) are examples of image data and determination result patterns corresponding to the image data. It is a flowchart explaining the main process. It is a flowchart explaining the marker color setting process. It is a flowchart explaining the determination target area setting process. It is a flowchart explaining the correspondence information registration process. It is a flowchart explaining the determination result table generation process. It is a flowchart explaining the marker position detection process. It is a flowchart explaining the work monitoring process. This is another example of the judgment result table. It is a figure which shows the main function which the work monitoring apparatus of 2nd Embodiment has. It is a flowchart explaining the work monitoring process of 2nd Embodiment. It is a figure which shows the main function which the work monitoring apparatus of 3rd Embodiment has. It is a flowchart explaining the work monitoring process of 3rd Embodiment. It is a figure which shows the main function which the work monitoring apparatus of 4th Embodiment has. It is a flowchart explaining the work monitoring process of 4th Embodiment. It is a figure which shows the example of the period to be analyzed.

Hereinafter, the embodiment will be described with reference to the drawings. In the following description, the same or similar configurations may be designated by the same reference numerals and duplicate description may be omitted. Further, in the following description, when it is necessary to distinguish the same type of configuration, an identifier (number, alphabet, etc.) may be written in parentheses after the code that collectively refers to the configuration.

[First Embodiment]
FIG. 1 shows a schematic configuration of the work monitoring system 1 described as the first embodiment. As shown in the figure, the work monitoring system 1 includes an image acquisition device 3, various sensors 4, and a work monitoring device 100 which is an information processing device. The image acquisition device 3, various sensors 4, and the work monitoring device 100 are communicably connected via a wired or wireless communication means 5. The configuration of the communication means 5 is not necessarily limited, but for example, a communication means compliant with various communication standards such as USB (Universal Serial Bus) and RS-232C, LAN (Local Area Network), WAN (Wide Area Network), the Internet, and the like. Dedicated line, etc.

The image acquisition device 3 (shooting device) is a device that acquires image data of the worker 2 and the surroundings of the worker 2, and is, for example, a camera (digital camera) that acquires (shoots) image data of a moving image or a still image. (RGB camera), infrared camera, thermography camera, time of flight (TOF: Time Of Flight) camera, stereo camera, etc.).

Various sensors 4 are provided in the work environment in which the worker 2 works, and output physical information about the worker 2 and the work environment. The various sensors 4 include, for example, a motion detection sensor, a human sensor, a temperature sensor, a humidity sensor, an acceleration sensor, a speed sensor, an acoustic sensor (microphone), an ultrasonic sensor, a vibration sensor, a millimeter wave radar, and a laser radar (LIDAR: Laser). Imaging Detection and Langing), an infrared depth sensor.

The work monitoring device 100 performs processing related to monitoring the work of the worker 2 based on the image data acquired by the image acquisition device 3.

FIG. 2 shows an example of the hardware configuration of the work monitoring device 100. The work monitoring device 100 is an information processing device (computer), and includes a processor 11, a main storage device 12, an auxiliary storage device 13, an input device 14, an output device 15, and a communication device 16.

The processor 11 is, for example, a device that performs arithmetic processing, such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), and an AI (Artificial Intelligence) chip. The main storage device 12 is a device that stores programs and data, and is, for example, a ROM (Read Only Memory) (SRAM (Static Random Access Memory), NVRAM (Non Volatile RAM), mask ROM (Mask Read Only Memory), PROM. (Programmable ROM), etc.), RAM (RandomAccessMemory) (DRAM (DynamicRandomAccessMemory), etc.), etc. The auxiliary storage device 13 is a hard disk drive (Hard Disk Drive), a flash memory (Flash Memory), an SSD (Solid State Drive), an optical storage device (CD (Compact Disc), DVD (Digital Versatile Disc), etc.) and the like. .. The programs and data stored in the auxiliary storage device 13 are read into the main storage device 12 at any time.

The input device 14 is a user interface that receives information from the user, and is, for example, a keyboard, a mouse, a card reader, a touch panel, or the like. The output device 15 is a user interface that outputs various information (display output, audio output, print output, etc.). For example, a display device (LCD (Liquid Crystal Display), graphic card, etc.) that visualizes various information, audio, etc. Output devices (speakers), printing devices, etc.

The communication device 16 is a communication interface that communicates with another device via the communication means 5, and is, for example, a NIC (Network Interface Card), a wireless communication module, a USB (Universal Serial Interface) module, a serial communication module, or the like. The communication device 16 can also function as an input device that receives information from another device that is communicably connected. The communication device 16 can also function as an output device that transmits information to other devices that are communicably connected. The work monitoring device 100 communicates with the image acquisition device 3 and various sensors 4 via the communication means 5 by the communication device 16.

The various functions included in the work monitoring device 100 are provided by the processor 11 reading and executing the program stored in the main storage device 12, or by the hardware (FPGA, ASIC) constituting the work monitoring device 100. , AI chip, etc.).

FIG. 3 shows the main functions of the work monitoring device 100. As shown in the figure, the work monitoring device 100 includes a storage unit 110, a data acquisition unit 120, a preparation setting processing unit 130, a determination result table generation unit 140, and a monitoring processing unit 150. In addition to the above functions, the work monitoring device 100 may further include functions such as an operating system, a device driver, a file system, and a DBMS (DataBase Management System).

Among the above functions, the storage unit 110 stores image data 111, sensor data 112, marker color data 113, determination target area data 114, determination result table 115, and determination result pattern / work content correspondence information 116. The storage unit 110 stores such information (data) as, for example, a database table provided by the DBMS or a file provided by the file system.

The image data 111 is data acquired from the image acquisition device 3, and is, for example, frame data that constitutes still image data or moving image data sent from the image acquisition device 3. The sensor data 112 is data acquired from various sensors 4. The details of the marker color data 113, the determination target area data 114, the determination result table 115, and the determination result pattern / work content correspondence information 116 will be described later.

The data acquisition unit 120 acquires (receives) data sent from another device. As shown in the figure, the data acquisition unit 120 includes an image data acquisition unit 121 and a sensor data acquisition unit 122. Of these, the image data acquisition unit 121 acquires image data 111 from the image acquisition device 3. The sensor data acquisition unit 122 acquires sensor data 112 from various sensors 4.

The preparation setting processing unit 130 performs processing related to setting various information used when the work monitoring device 100 monitors the work of the worker 2. As shown in the figure, the preparation setting processing unit 130 includes a marker color setting processing unit 131, a determination target area setting processing unit 132, and a corresponding information setting processing unit 133.

Of these, the marker color setting processing unit 131 is provided on the body of the worker 2, the work object, the tool used for the work, and the like in order to recognize the position of the worker 2 and the object based on the image data 111. (Hereinafter referred to as "marker"), the color is set. For example, the marker color setting processing unit 131 acquires color data (for example, data expressed by RGB values) in the area where the marker is reflected in the image data 111, and refers to the identifier of the marker (hereinafter, referred to as “marker ID”). ) And the color data are associated with each other to generate marker color data 113, which is data including information.

The determination target area setting processing unit 132 performs processing related to setting an area (hereinafter, referred to as “determination target area”) for the image data 111. The determination target area setting processing unit 132 receives, for example, the setting of the determination target area from the user. Further, the determination target area setting processing unit 132 automatically sets the determination target area in the image data. The determination target area setting processing unit 132 generates the determination target area data 114, which is data including information for specifying the set determination target area. The determination target area data 114 includes, for example, information in which an identifier of the determination target area (hereinafter, referred to as “determination target area ID”) and information indicating the area in the coordinate system of the image data are associated with each other.

In the correspondence information setting processing unit 133, an object provided with a marker (a predetermined part of the body of the worker 2, a work target, an object related to work such as a tool used by the worker 2) exists in each determination target area ( The pattern of the judgment result table 115 (hereinafter, referred to as "judgment result pattern"), which is data including information indicating whether or not the judgment result is reflected, and information for specifying the work content corresponding to the judgment result pattern (hereinafter, referred to as "judgment result pattern") It is referred to as "work content information") and is associated with the determination result pattern / work content correspondence information 116. The work content information includes information indicating whether the work of the determination result pattern is for normal work or abnormal work (hereinafter, referred to as "normal / abnormal classification"). .. The details of the determination result table 115 will be described later.

The determination result table generation unit 140 generates the determination result table 115 based on the image data 111. As shown in the figure, the determination result table generation unit 140 includes a marker position detection processing unit 141 and a determination result table generation processing unit 142.

Of these, the marker position detection processing unit 141 detects the position where the marker is reflected in the image data 111 (in this example, the position of the center of gravity of the region where the marker is moved) using the marker color data 113.

The judgment result table generation processing unit 142 generates the judgment result table 115 by comparing the position of the detected marker with the judgment target area data 114.

FIG. 4A shows an example of image data 111. In the illustrated image data 111, a plurality of markers 42 (1) to 42 (4) provided in each of the objects 41 (1) to 41 (4) are shown. Further, the determination target areas 43 (1) to (3) are set.

FIG. 4B is drawn by extracting the positions of the markers 42 (1) to (4) and the determination target areas 43 (1) to (3) shown in the image data 111 shown in FIG. 4A. It is a figure. In this example, the determination target areas 43 (1) to (3) have a rectangular frame, but other shapes may be used.

FIG. 4C is a determination result table 115 generated based on the image data 111 shown in FIG. 4A. As shown in the figure, the determination result table 115 is a two-dimensional table in which the marker ID is set in the row direction and the determination target area ID is set in the column direction. Further, as shown in the figure, an item (hereinafter, referred to as “Visible”) indicating visible / invisible (whether or not the marker is reflected in the image data 111) of the marker is further provided in the column direction. The determination result pattern described above has the same configuration as the determination result table 115.

In the case of this example, for example, the object whose marker ID is "Object 1" is reflected in the image data 111, so the value of "Visible" is set to "True" and the judgment target area ID is "Area 1". Since it is within the area, the item is set to "True", and since the judgment target area ID is outside the area of "Area 2", the item is set to "False" and the judgment target area ID is "Area 3". Since it is outside the area, the item is set to "False".

Further, since the object whose marker ID is "Object 2" is reflected in the image data 111, the value of "Visible" is set to "True", and the judgment target area ID is within the judgment target area of "Area 1". Therefore, the item is set to "True", and the judgment target area ID is within the judgment target area of "Area 2", so the item is set to "True", and the judgment target area ID is the judgment target of "Area 3". Since it is outside the area, the item is set to "False".

Further, the object whose marker ID is "Object 3" is reflected in the image data 111, so the value of "Visible" is set to "True", and the judgment target area ID is outside the judgment target area of "Area 1". Therefore, the item is set to "False" and the judgment target area ID is outside the judgment target area of "Area 2", so the item is set to "False" and the judgment target area ID is "Area 3". Since it is in the area, the item is set to "True".

Further, the object whose marker ID is "Object 4" is reflected in the image data 111, so the value of "Visible" is set to "True", and the judgment target area ID is outside the judgment target area of "Area 1". Therefore, the item is set to "False" and the judgment target area ID is outside the judgment target area of "Area 2", so the item is set to "False" and the judgment target area ID is "Area 3". Since it is outside the area, the item is set to "False".

Further, since the object whose marker ID is "Object 5" is not reflected in the image data 111, the value of "Visible" is set to "False", and the judgment target area IDs are "Area 1", "Area 2", and so on. The value of each item of "Area 3" is set to "False".

The contents of the determination result table 115 shown above represent the characteristics of the individual work performed by the worker 2. Therefore, by preparing in advance the judgment result table 115 based on the image data 111 acquired for each work (normal work or abnormal work) performed by the worker 2, the worker 2 is newly prepared. The work performed by the worker 2 can be easily and accurately specified based on the image data 111 reflecting the work of.

FIG. 5 shows an example of a judgment result pattern (judgment result table 115) generated for each work performed by the worker 2 (three cases shown by (a) to (c) in the figure). Since the determination result table 115 has an item of "Visible" and includes information on whether or not the marker is reflected in the image data 111, the work of the worker 2 can be specified with high accuracy. .. Further, since the position of the marker displayed in the image data 111 can be automatically specified by using the information processing device, the user can easily set the judgment target area in the image data 111 and easily perform the judgment result table. 115 can be produced. As described above, according to the work monitoring device 100 of the present embodiment, it is possible to easily realize a mechanism for monitoring the work of the worker 2 at each site where different work is performed.

Returning to FIG. 3, the monitoring processing unit 150 monitors the work of the worker 2 based on the image data 111 sent from the image acquisition device 3. As shown in the figure, the monitoring processing unit 150 includes a determination result table acquisition unit 151 and an abnormality presence / absence determination processing unit 152.

Of these, the determination result table acquisition unit 151 acquires the determination result table 115 generated based on the image data 111 from the determination result table generation unit 140.

The abnormality presence / absence determination processing unit 152 determines whether the work of the worker 2 is normal or abnormal based on the determination result table 115, and outputs the determination result. In the determination result table 115 generated based on the image data 111, for example, the abnormality presence / absence determination processing unit 152 sets the normal / abnormal classification registered in the determination result pattern / work content correspondence information 116 to "normal". If it matches any of the determination result patterns, it is determined that the work of the worker 2 is normal, and if it does not match any of the determination result patterns, the work of the worker 2 is determined to be abnormal. Further, in the abnormality presence / absence determination processing unit 152, for example, the determination result table 115 generated based on the image data 111 is registered in the determination result pattern / work content correspondence information 116, and the normal / abnormal classification is "abnormal". If it matches any of the above, it is determined that the work of the worker 2 is abnormal, and if it does not match any of the above, it is determined that the work of the worker 2 is normal. Note that the match between the determination result table 115 and the determination result pattern does not necessarily mean an exact match, and for example, if the degree of mismatch is within a preset allowable range, it may be determined as a match.

<Processing explanation>
Subsequently, the processing performed by the work monitoring device 100 will be described. In the following description, it is assumed that the shooting range of the image acquisition device 3 is fixed while the work monitoring device 100 monitors the work of the worker 2. Also, each marker shall have a different color.

FIG. 6 is a flowchart illustrating a process (hereinafter, referred to as “main process S600”) performed by the work monitoring device 100 when monitoring the work of the worker 2 at the site.

As shown in the figure, first, the preparation setting processing unit 130 sets the information necessary for monitoring the work of the worker 2 (marker color setting process S611, determination target area setting process S612, and correspondence information registration). Process S613) is performed. Details of each of these processes will be described later.

Subsequently, the monitoring processing unit 150 performs a process of monitoring the work of the worker 2 (work monitoring process S614).

FIG. 7 is a flowchart illustrating the details of the marker color setting process S611 of FIG.

As shown in the figure, first, the image data acquisition unit 121 acquires the image data 111 from the image acquisition device 3 (S711).

Subsequently, the marker color setting processing unit 131 acquires the color data (RGB data) of the marker area reflected in the image data 111 (S712). When a plurality of markers are displayed in the image data 111, the marker color setting processing unit 131 acquires the color data of the area of each marker.

Subsequently, the marker color setting processing unit 131 converts the acquired color data into data in a predetermined color space (HSV (Hue Saturation Value), HSB (Hue Saturation Brightness), etc.) (S713). It should be noted that the conversion to the color space does not necessarily have to be performed, and whether or not the conversion is performed is determined according to, for example, the required monitoring accuracy (the accuracy of determining whether the work of the worker 2 is normal or abnormal). Just do it. The same applies to which color space is selected.

Subsequently, the marker color setting processing unit 131 associates the converted data with the marker ID and stores it as the marker color data 113 (S714).

FIG. 8 is a flowchart illustrating the details of the determination target area setting process S612 of FIG.

As shown in the figure, first, the image data acquisition unit 121 acquires the image data 111 from the image acquisition device 3 (S811).

Subsequently, the determination target area setting processing unit 132 sets one or more determination target areas in the acquired image data 111 (S812). The determination target area setting processing unit 132 sets the determination target area by, for example, interactive processing with the user. The user appropriately sets the determination target area so as to secure the required monitoring accuracy based on the relationship between the setting contents of the determination target area in the past and the determination accuracy. Further, the determination target area setting processing unit 132 automatically sets the determination target area so that the required monitoring accuracy is ensured, for example, based on the relationship between the setting contents of the past determination target area and the determination accuracy.

Subsequently, the determination target area setting processing unit 132 stores the set determination target area as the determination target area data 114 (S813).

FIG. 9 is a flowchart illustrating the details of the corresponding information registration process S613 of FIG.

As shown in the figure, first, the image data acquisition unit 121 acquires the image data 111 from the image acquisition device 3 (S911).

Subsequently, the correspondence information setting processing unit 133 sends the acquired image data 111 to the determination result table generation unit 140. The determination result table generation unit 140 performs a process of acquiring the determination result table 115 based on the sent image data 111 (hereinafter, referred to as "determination result table generation process S912"), and generates the determination result table 115. It is returned to the correspondence information setting processing unit 133. The details of the determination result table generation process S912 will be described later.

Subsequently, the corresponding information setting processing unit 133 accepts the input (setting) of the work content information for the image data 111 from the user (S913).

Subsequently, the correspondence information setting processing unit 133 associates the judgment result table 115 sent from the judgment result table generation unit 140 with the received work content information and registers it in the judgment result pattern / work content correspondence information 154 ( S914).

FIG. 10 is a flowchart illustrating the details of the determination result table generation process S912 of FIG.

As shown in the figure, first, the marker position detection processing unit 141 performs a process of detecting the position of each marker included in the image data 111 (hereinafter, referred to as “marker position detection process S1011”). The details of the marker position detection process S1011 will be described later.

Subsequently, the determination result table generation processing unit 142 acquires the determination target area data 114 (S1012).

Subsequently, the determination result table generation processing unit 142 generates the determination result table 115 by comparing the position of the detected marker with the acquired determination target area data 114 (S1013).

FIG. 11 is a flowchart illustrating the details of the marker position detection process S811 of FIG.

As shown in the figure, first, the marker position detection processing unit 141 acquires the marker color data 113 (S1111). When the marker color data 113 of a plurality of markers exists, the marker position detection processing unit 141 acquires all the marker color data 113.

Subsequently, the marker position detection processing unit 141 transfers the area corresponding to the marker color data 113 from the image data 111 (a region in which the match or similarity is equal to or less than a preset threshold value; hereinafter referred to as a “marker area”). Is detected (S1112). When the marker color data 113 of a plurality of markers is acquired, the marker region is detected for each marker color data.

Subsequently, the marker position detection processing unit 141 removes noise information from the detected marker region by using, for example, a known noise removal technique (image processing, morphology calculation, etc.) (S1113).

Subsequently, the marker position detection processing unit 141 specifies the position of the center of gravity (coordinates of the center of gravity) of the marker region (S1114).

Subsequently, the marker position detection processing unit 141 checks whether or not there are a plurality of marker regions corresponding to the same marker color data, and if there are a plurality of marker regions, one of them is selected (S1115). For example, the marker position detection processing unit 141 selects the color of the marker region closest to the marker color data. Further, for example, the marker position detection processing unit 141 selects the marker area based on whether or not the position of the marker area in the image data 111 is within a predetermined range.

Subsequently, the marker position detection processing unit 141 sets the center of gravity of the selected marker area as the marker position (S1116).

FIG. 12 is a flowchart illustrating the details of the work monitoring process S614 of FIG.

As shown in the figure, first, the image data acquisition unit 121 acquires one or more image data 111 from the image acquisition device 3 (S1211). The image data 111 is, for example, a one-frame image of moving image data showing the state of the worker 2.

Subsequently, the determination result table acquisition unit 151 passes the acquired image data 111 to the determination result table generation unit 140, and acquires the determination result table 115 from the determination result table generation unit 140 (S1212). Since the process of S1212 is the same as the determination result table generation process S912 described above, the description thereof will be omitted.

Subsequently, the abnormality presence / absence determination processing unit 152 determines whether or not the work of the worker 2 is normal based on the acquired determination result table 115 (S1213), and outputs the determination result to the output device 15 (S1214).

Subsequently, the monitoring processing unit 150 determines whether or not the work of the worker 2 has been completed (S1215). If the work is completed (S1215: YES), the monitoring process S414 is completed, and if the work is not completed (S1215: NO), the process returns to S1211. The monitoring processing unit 150 determines whether or not the work has been completed, for example, based on the information input from the user.

As described in detail above, according to the work monitoring device 100 of the present embodiment, by preparing the determination result pattern in advance, the work can be performed from the image data 111 that newly reflects the work of the worker 2. It is possible to accurately determine whether the work performed by the person 2 is normal or abnormal. Therefore, it is possible to easily realize a mechanism for monitoring the work of the worker 2 at each site.

By the way, the determination result table 115 shown above includes information on the marker (information on whether or not the position of the marker is within the determination target area and information on whether or not the marker is reflected in the image data 111). However, the determination result table 115 and the determination result pattern further include the information of the sensor data acquired from the various sensors 4 when the image data 111 is acquired, and the monitoring processing unit 150 also considers the value of the sensor data. The determination result table 115 may be compared with the determination result pattern / work content correspondence information 116 to identify the work and determine whether or not there is an abnormality in the work. By further including the sensor data information in the determination result table 115 and the determination result pattern, the characteristics of each work can be expressed in more detail, and the determination accuracy of the work can be improved.

FIG. 13 shows an example of the judgment result table 115 in that case. In the determination result table 115 illustrated, in addition to each item of the determination result table 115 described above, an item (“Sensor 1”, “Sensor 1”, “Sensor 1”, “Sensor 1”, “ Sensor 2 ”) is included. It also includes items (“Value 1”, “Value 2”) for which sensor data values are set in the column direction.

[Second Embodiment]
The work monitoring device 100 of the second embodiment uses the work monitoring mechanism of the worker 2 in the first embodiment to determine the time required for the work by the worker 2 (hereinafter, referred to as “working time”). The work performed by the worker 2 is monitored (determines whether the work is normal or abnormal) by comparing the acquired work time with the preset standard work time. Hereinafter, the parts different from those of the first embodiment will be mainly described.

FIG. 14 shows the main functions of the work monitoring device 100 of the second embodiment. As shown in the figure, the storage unit 110 of the work monitoring device 100 of the second embodiment has each work performed by the worker 2 in addition to the information stored by the storage unit 110 of the work monitoring device 100 of the first embodiment. The standard working time 117 for each work, which is data including information indicating the standard working time of the above, is stored. Further, the abnormality presence / absence determination processing unit 152 of the work monitoring device 100 of the second embodiment has a function of determining whether the work performed by the worker 2 is normal or abnormal by comparing the standard work time with the work time.

FIG. 15 is a flowchart illustrating the work monitoring process S614 in the second embodiment. Since the other processes in the second embodiment (the other processes described in the first embodiment performed by the work monitoring device 100) are the same as those in the first embodiment, the description thereof will be omitted.

As shown in the figure, first, the monitoring processing unit 150 initializes the work variables "working time" and "previous work" used in the following processing (S1511). In the "working time", the cumulative working time when the worker 2 is performing a certain work is set. Further, in the "previous work", information indicating the work specified last time in the loop processing from S1512 (for example, the above-mentioned work content information) is set.

Subsequently, the image data acquisition unit 121 acquires one or more image data 111 from the image acquisition device 3 (S1512). The image data 111 is a one-frame image of moving image data showing the state of the worker 2. In the loop processing of S1512 to S1517, the image data 111 of one frame of the moving image data is sequentially selected in chronological order.

Subsequently, the determination result table acquisition unit 151 passes the acquired image data 111 to the determination result table generation unit 140, and acquires the determination result table 115 from the determination result table generation unit 140 (S1513). Since the process of S1513 is the same as the determination result table generation process S912 described in the first embodiment, the description thereof will be omitted.

Subsequently, the abnormality presence / absence determination processing unit 152 identifies the work performed by the worker 2 by comparing the acquired determination result table 115 with the determination result pattern / work content correspondence information 116 (S1514).

Subsequently, the abnormality presence / absence determination processing unit 152 determines whether or not the specified work is the same as the "previous work" (S1515). If the specified work is the same as the "previous work" (S1515: YES), the process proceeds to S1516. In the first loop, the process of S1515 is skipped and the process proceeds to S1516.

If the specified work is different from the "previous work" (S1515: NO), the process proceeds to S1518.

In S1516, the abnormality presence / absence determination processing unit 152 updates the "working time" (for example, the time from the previous update to the present is added).

In S1517, the abnormality presence / absence determination processing unit 152 sets the work specified in S1514 (for example, work content information) to "previous work". After that, the process returns to S1512.

In S1518, the abnormality presence / absence determination processing unit 152 acquires the standard work time of the work specified in S1514 from the determination result pattern / work content correspondence information 116, compares the “work time” with the standard work time, and compares the “work time” with the standard work time. Determines if the work being done by is abnormal. For example, if the difference between the "working time" and the standard working time is equal to or less than the predetermined time, the abnormality presence / absence determination processing unit 152 determines that the work performed by the worker 2 is normal, and the difference exceeds the predetermined time. If so, the work performed by the worker 2 is determined to be abnormal. By appropriately setting the predetermined time, for example, the determination may be made while considering the time spent by the worker 2 in addition to the work such as the break time.

In S1519, the abnormality presence / absence determination processing unit 152 outputs the determination result to the output device 15.

As described above, according to the above mechanism, by using the work monitoring mechanism of the worker 2 in the first embodiment, the work performed by the worker 2 based on the work time of the work performed by the worker 2. It is possible to realize a mechanism for monitoring whether or not is normal or abnormal.

[Third Embodiment]
The work monitoring device 100 of the third embodiment monitors whether or not the worker 2 is performing the work according to the correct work order by using the work monitoring mechanism of the worker 2 in the first embodiment. Hereinafter, the parts different from those of the first embodiment will be mainly described.

FIG. 16 shows the main functions of the work monitoring device 100 of the third embodiment. As shown in the figure, the storage unit 110 of the work monitoring device 100 of the third embodiment stores information indicating the correct work order in addition to the information stored by the storage unit 110 of the work monitoring device 100 of the first embodiment. The work order information 118, which is the data to be included, is stored. Further, the abnormality presence / absence determination processing unit 152 of the work monitoring device 100 of the third embodiment has a function of determining whether or not the worker 2 is performing work according to a preset regular work order.

FIG. 17 is a flowchart illustrating the work monitoring process S614 according to the third embodiment. Since the other processes in the third embodiment (the other processes described in the first embodiment performed by the work monitoring device 100) are the same as those in the first embodiment, the description thereof will be omitted.

As shown in the figure, first, the monitoring processing unit 150 initializes the "previous work" which is a work variable used in the following processing (S1711). In the "previous work", information indicating the work specified last time in the loop process from S1712 (for example, the above-mentioned work content information) is set.

Subsequently, the image data acquisition unit 121 acquires one or more image data 111 from the image acquisition device 3 (S1712). The image data 111 is a one-frame image of moving image data showing the state of the worker 2. In the loop processing of S1712 to S1716, the image data 111 of one frame of the moving image data is sequentially selected in chronological order.

Subsequently, the determination result table acquisition unit 151 passes the acquired image data 111 to the determination result table generation unit 140, and acquires the determination result table 115 from the determination result table generation unit 140 (S1713). Since the process of S1713 is the same as the determination result table generation process S912 described in the first embodiment, the description thereof will be omitted.

Subsequently, the abnormality presence / absence determination processing unit 152 identifies the work performed by the worker 2 by comparing the acquired determination result table 115 with the determination result pattern / work content correspondence information 116 (S1714).

Subsequently, the abnormality presence / absence determination processing unit 152 compares the specified work, the "previous work", and the work order information 118 to determine whether the specified work is a work according to the work order in the work order information 118. (S1715). In the first loop, the process of S1715 is skipped and the process proceeds to S1716. If the specified work follows the work order in the work order information 118 (S1715: YES), the process proceeds to S1716. If the specified work is not a work according to the work order in the work order information 118 (S1715: NO), the process proceeds to S1717.

In S1716, the abnormality presence / absence determination processing unit 152 sets the work specified in S1714 (for example, work content information) to "previous work". After that, the process returns to S1712.

In S1717, the abnormality presence / absence determination processing unit 152 outputs the determination result (information indicating that the work order does not follow the correct work order) to the output device 15.

By using the work monitoring mechanism of the worker 2 in the first embodiment in this way, it is possible to realize a mechanism for monitoring whether or not the work performed by the worker 2 follows the correct work order.

[Fourth Embodiment]
The work monitoring device 100 of the fourth embodiment uses the work monitoring mechanism of the worker 2 in the first embodiment to set the period during which the work to be analyzed is performed among the periods performed by the worker 2. Identify. Hereinafter, the parts different from those of the first embodiment will be mainly described.

FIG. 18 shows the main functions of the work monitoring device 100 of the fourth embodiment. As shown in the figure, the storage unit 110 of the work monitoring device 100 of the fourth embodiment shows the work to be analyzed in addition to the information stored by the storage unit 110 of the work monitoring device 100 of the first embodiment. The analysis target work information 119, which is data including information (for example, the above-mentioned work content information), is stored. Further, the abnormality presence / absence determination processing unit 152 of the work monitoring device 100 of the fourth embodiment has a function of specifying the period during which the work to be analyzed is being performed among the periods during which the worker 2 is performing.

FIG. 19 is a flowchart illustrating the work monitoring process S614 according to the fourth embodiment. Since the other processes in the fourth embodiment (the other processes described in the first embodiment performed by the work monitoring device 100) are the same as those in the first embodiment, the description thereof will be omitted.

As shown in the figure, first, the image data acquisition unit 121 acquires one or more image data 111 from the image acquisition device 3 (S1911). The image data 111 is a one-frame image of moving image data showing the state of the worker 2. In the loop processing of S1911 to S1916, the image data 111 of one frame of the moving image data is sequentially selected in chronological order.

Subsequently, the determination result table acquisition unit 151 passes the acquired image data 111 to the determination result table generation unit 140, and acquires the determination result table 115 from the determination result table generation unit 140 (S1912). Since the process of S1912 is the same as the determination result table generation process S912 described in the first embodiment, the description thereof will be omitted.

Subsequently, the abnormality presence / absence determination processing unit 152 identifies the work performed by the worker 2 by comparing the acquired determination result table 115 with the determination result pattern / work content correspondence information 116 (S1913).

Subsequently, the abnormality presence / absence determination processing unit 152 compares the specified work with the analysis target work information 119, and determines whether or not the specified work is the analysis target (S1914). If the identified work is the analysis target (S1914: YES), the process proceeds to S1915. If the specified work is not the analysis target (S1914: NO), the process returns to S1911.

In S1915, the abnormality presence / absence determination processing unit 152 outputs information indicating that the worker 2 is performing the work to be analyzed at the timing of the image data to the output device 15.

According to the above mechanism, when the worker 2 continuously performs a plurality of works, it is possible to output information indicating the period during which each work is performed. For example, the work performed by the worker 2 can be analyzed. A person who intends to analyze (hereinafter referred to as an "analyzer") can easily specify the period to be analyzed.

FIG. 20 shows an example of the above period that the analyst analyzes. For example, as shown in (a), the analyst can specify the start time point and the end time point of the work to be analyzed by using the above mechanism. Further, for example, as shown in (b), the analyst can specify the start time to be analyzed by using the above mechanism, and can specify a predetermined time from the start time as the period to be analyzed. Further, for example, as shown in (c), the analyst uses the above mechanism to specify the end time point to be analyzed, and the period from the end time point to the end time point is the period to be analyzed. Can be specified as.

Although one embodiment of the present invention has been described in detail above, it goes without saying that the present invention is not limited to the above embodiment and can be variously modified without departing from the gist thereof. For example, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to those having all the described configurations. Further, it is possible to add, delete, or replace a part of the configuration of the above embodiment with another configuration.

For example, the work monitoring process S614 described in each of the above embodiments may be executed for the moving image data acquired in real time, or may be executed ex post facto for the recorded moving image data.

Further, each of the above configurations, functional units, processing units, processing means, etc. may be realized by hardware by designing a part or all of them by, for example, an integrated circuit. Further, each of the above configurations, functions, etc. may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files that realize each function can be placed in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

Also, in each of the above figures, the control lines and information lines are shown as necessary for explanation, and not all the control lines and information lines in the implementation are necessarily shown. For example, in practice almost all configurations may be considered interconnected.

The arrangement form of various functional units, various processing units, and various databases of each information processing device described above is only an example. The arrangement form of various function units, various processing units, and various databases can be changed to the optimum arrangement form from the viewpoint of the performance, processing efficiency, communication efficiency, and the like of the hardware and software included in these devices.

In addition, the configuration of the database (schema, etc.) that stores the various data described above can be flexibly changed from the viewpoints of efficient use of resources, improvement of processing efficiency, improvement of access efficiency, improvement of search efficiency, and the like.

Further, instead of the position detection method using the marker described above, the position of the object may be detected by using a sensor or an image recognition method that can automatically detect the body of the worker 2. Further, the position of the object may be detected by using a sensor or an image recognition method that can automatically detect the work object or the tool used for the work. In this case, the same processing as that of the above-described embodiment is performed using the automatically detected position information of the body, the work object, and the like.

1 Work monitoring system 2 Worker 3 Image acquisition device 4 Various sensors 5 Communication means 100 Work monitoring device 111 Image data 112 Sensor data 113 Marker color data 114 Judgment target area data 115 Judgment result table 116 Judgment result pattern / work content correspondence information 120 Data acquisition unit 121 Image data acquisition unit 122 Sensor data acquisition unit 130 Preparation setting processing unit 131 Marker color setting processing unit 132 Judgment target area setting processing unit 133 Corresponding information setting processing unit 140 Judgment result table generation unit 141 Marker position detection processing unit 142 Judgment result table generation processing unit 150 Monitoring processing unit 151 Judgment result table acquisition unit 152 Abnormality presence / absence judgment processing unit S600 Main processing S611 Marker color setting processing S612 Judgment target area setting processing S613 Corresponding information registration processing S912 Judgment result table generation processing S1011 Marker position Detection process S614 Work monitoring process

Claims (15)

1. It is configured using an information processing device with a processor and memory.
   An image data acquisition unit that acquires image data that shows the worker performing work,
   An object position detection unit that acquires information indicating each position in the image data of a plurality of objects reflected in the image data, and an object position detection unit.
   A determination result table including information indicating the result of determining whether or not each of the positions of the plurality of objects is included in each of the one or more determination target areas which are the areas set in the image data. Judgment result table generator to be generated and
   A storage unit that stores a determination result pattern / work content correspondence information, which is information in which the determination result table generated based on the image data is associated with work content information which is information indicating work in the image data, and a storage unit.
   The judgment result table is generated for the newly acquired image data, and the work performed by the worker is specified by comparing the generated judgment result table with the judgment result pattern / work content correspondence information. Monitoring processing unit and
   A work monitoring device equipped with.
2. The work monitoring device according to claim 1.
   The determination result table further includes information indicating whether or not the position of the object is included in the image data.
   Work monitoring device.
3. The work monitoring device according to claim 1.
   Further provided with a sensor data acquisition unit that acquires sensor data, which is data output by a sensor provided at the site where the worker performs work.
   The determination result table further includes the sensor data acquired from the sensor.
   The monitoring processing unit generates the determination result table based on the newly acquired image data and the sensor data acquired from the sensor at the time of acquisition of the image data, and the generated determination result table and the determination result. Identify the work that a person is doing by comparing it with the pattern / work content correspondence information.
   Work monitoring device.
4. The work monitoring device according to any one of claims 1 to 3.
   The storage unit stores information indicating a standard working time required for the predetermined work, and stores the information.
   The image data acquisition unit acquires time-series image data showing the worker performing the work, and obtains the image data.
   The determination result table generation unit generates the determination result table for each of the image data in the time series.
   The monitoring processing unit specifies the work performed by the worker by comparing each of the judgment result tables with the judgment result pattern / work content correspondence information, so that the worker can perform the predetermined work. The work performed by the worker is monitored by acquiring the required work time and comparing the work time with the standard work time.
   Work monitoring device.
5. The work monitoring device according to any one of claims 1 to 3.
   The storage unit stores information indicating a standard work order for a plurality of the work performed by the worker.
   The image data acquisition unit acquires time-series image data showing the worker performing the work, and obtains the image data.
   The determination result table generation unit generates the determination result table for each of the image data in the time series.
   The monitoring processing unit identifies the work performed by the worker by comparing each of the judgment result tables with the judgment result pattern / work content correspondence information, thereby performing the work performed by the worker. The work performed by the worker is monitored by specifying the order and comparing the specified work order with the standard work order.
   Work monitoring device.
6. The work monitoring device according to any one of claims 1 to 3.
   The storage unit stores information indicating a predetermined work performed by the worker, and stores information indicating the predetermined work.
   The image data acquisition unit acquires time-series image data showing the worker performing the work, and obtains the image data.
   The determination result table generation unit generates the determination result table for each of the image data in the time series.
   The monitoring processing unit determines the work performed by the worker by comparing each of the judgment result tables with the judgment result pattern / work content correspondence information, thereby starting or ending the predetermined work. Generate timing information,
   Work monitoring device.
7. The work monitoring device according to claim 1.
   A determination target area setting processing unit that accepts the setting of the determination target area for the image data is further provided.
   Work monitoring device.
8. The work monitoring device according to claim 1.
   The object is at least one of a part of the worker's body, a work object, and a tool used by the worker in his work.
   Work monitoring device.
9. The work monitoring device according to claim 1.
   The object position detection unit acquires information indicating the position of the object in the image data by recognizing an image of a marker provided on the object.
   Work monitoring device.
10. An information processing device with a processor and memory
    Acquire image data showing how the worker is working,
    Information indicating the position of each of the plurality of objects reflected in the image data in the image data is acquired.
    A determination result table including information indicating the result of determining whether or not each of the positions of the plurality of objects is included in each of the one or more determination target areas which are the areas set in the image data. Generate and
    The judgment result pattern / work content correspondence information, which is information in which the determination result table generated based on the image data is associated with the work content information which is information indicating the work in the image data, is stored.
    The judgment result table is generated for the newly acquired image data, and the work performed by the worker is specified by comparing the generated judgment result table with the judgment result pattern / work content correspondence information. ,
    Work monitoring method.
11. The work monitoring method according to claim 10.
    The determination result table further includes information indicating whether or not the position of the object is included in the image data.
    Work monitoring method.
12. The work monitoring method according to claim 10.
    The information processing device acquires sensor data, which is data output by a sensor provided at the site where the worker works.
    The determination result table further includes the sensor data acquired from the sensor.
    The information processing device generates the determination result table based on the newly acquired image data and the sensor data acquired from the sensor at the time of acquisition of the image data, and the generated determination result table and the determination result. Identify the work that a person is doing by comparing it with the pattern / work content correspondence information.
    Work monitoring method.
13. The work monitoring method according to any one of claims 10 to 12.
    The information processing device
    Stores information indicating the standard working time required for the predetermined work,
    Acquire time-series image data showing the worker performing the work,
    The determination result table is generated for each of the image data in the time series.
    By identifying the work performed by the worker by comparing each of the judgment result tables with the judgment result pattern / work content correspondence information, the work time required by the worker for the predetermined work is acquired. Then, the work performed by the worker is monitored by comparing the work time with the standard work time.
    Work monitoring method.
14. The work monitoring method according to any one of claims 10 to 12.
    The information processing device
    Stores information indicating a standard work sequence for a plurality of the work performed by the worker.
    Acquire time-series image data showing the worker performing the work,
    The determination result table is generated for each of the image data in the time series.
    By specifying the work performed by the worker by comparing each of the judgment result tables with the judgment result pattern / work content correspondence information, the order of the work performed by the worker is specified and specified. The work performed by the worker is monitored by comparing the work sequence performed with the standard work sequence.
    Work monitoring method.
15. The work monitoring method according to any one of claims 10 to 12.
    The information processing device
    Stores information indicating a predetermined work performed by the worker,
    Acquire time-series image data showing the worker performing the work,
    The determination result table is generated for each of the image data in the time series.
    By specifying the work performed by the worker by comparing each of the judgment result tables with the judgment result pattern / work content correspondence information, information indicating the start or end timing of the predetermined work is generated. To do
    Work monitoring method.

Images