JP7324618B2 - Work monitoring device and work monitoring method - Google Patents

Description

The present invention relates to a work monitoring device and a work monitoring method.

Patent Literature 1 describes a motion recognition method of a moving object for a robot to recognize the posture, motion, or behavior of a person. In the motion recognition method described above, eigenspace data A in which frame image data A for each basic motion of a moving body A are displayed as dots is created in advance and stored in a database, and frame image data B of a target moving body B is displayed as dots. The eigenspace data B and the eigenspace data A for each basic motion are compared, and the eigenspace data A having the closest distance from the eigenspace data B is selected to recognize the motion of the moving body B. Each frame image data A for each basic motion is obtained by making the moving body A perform the basic motion, photographing the moving body A performing the basic motion from multiple directions using a plurality of image input means, and acquiring each image input means. It is obtained from a compressed image created by superimposing a plurality of frame images after each weighting.

Patent Literature 2 describes a motion recognition system that recognizes the shape and motion of a hand in a user interface that uses human gestures and gestures. A motion recognition system recognizes the shape and motion of a target by processing time-series image data containing images of a specific target, extracts regions with motion from the time-series image data, and extracts the time-series image data. An area containing a color that characterizes the object is extracted from the image, and an area that is a moving area and contains a color that characterizes the object is extracted as an object area.

Non-Patent Document 1 describes various behavior recognition techniques that utilize sensors and cameras. In Non-Patent Document 2, "Zero
It describes that motion recognition is performed using "Shot Learning".

JP 2009-176059 A Japanese Patent Application Laid-Open No. 2001-16606

Michalis Vrigkas,Christophoros Nikou,Ioannis A Kakadiaris; Review of Human Activity Recognition Methods;Frontiers in Robotics and AI，16 November 2015 Kun Liu, Wu Liu, Huadong Ma, Wenbing Huang, Xiongxiong Dong; Generalized Zero-Shot Learning for Action Recognition with Web-Scale Video Data; Submitted on 20 October 2017; Cornell University

At product manufacturing sites, a mechanism has been introduced to monitor whether workers are following correct work procedures by recognizing the actions of workers for the purpose of quality control and ensuring the safety of workers. ing. Recently, technology using machine learning is attracting attention. For example, a publicly available database ("UCF101-Action Recognition Data Set
etc.) are being actively researched and developed.

By the way, since the work performed by workers differs from site to site, it is necessary to collect data on the various tasks performed by workers at each site when introducing a work monitoring system that uses motion recognition technology. . Moreover, even when using a database such as the one described above, it is necessary to collect data for each work site for unregistered work.

For example, the technique described in Patent Document 1 generates eigenspace data in advance from frame images for each basic motion, and performs motion recognition by comparing distances from motion features of the eigenspace. It is necessary to prepare a large amount of data in advance regarding basic operations, which places a heavy burden on on-site workers at the time of introduction.

On the other hand, the technique described in Patent Document 2 uses time-series image data to extract target regions based on motion regions and regions containing colors that characterize motion recognition targets, and performs motion recognition by analyzing the shapes. It is a method that does not require data in advance. However, since this technology recognizes motions based on shape, it is not possible to accurately recognize information about the position of the working hand and its surroundings.

In the technique described in Non-Patent Document 2, the label of data that does not appear in the learning data is predicted, but the data necessary for predicting the recognition target must be prepared on site.

The present invention has been made based on such a background, and provides a work monitoring device and a work monitoring method capable of accurately monitoring work performed by a worker without preparing a large amount of data in advance. for the purpose.

One of the present inventions for achieving the above object is a work monitoring device, which is constructed using an information processing device having a processor and a memory, and which displays image data showing a worker performing work. an image data acquisition unit for acquiring; an object position detection unit for acquiring information indicating respective positions in the image data of a plurality of objects appearing in the image data; and one area set in the image data. a determination result table generating unit for generating a determination result table containing information indicating a result of determining whether or not the position of each of the plurality of objects is included in each of the determination target regions; and the image data. a storage unit that stores determination result pattern/work content correspondence information that is information in which the determination result table generated based on the above is associated with work content information that is information indicating work in the image data; a monitoring processing unit that generates the determination result table for the image data obtained and identifies the work performed by the worker by comparing the generated determination result table with the determination result pattern/work content correspondence information; , wherein the determination result table further includes information indicating whether or not the position of the object is included in the image data.

In addition, the problems disclosed by the present application and their solutions will be clarified by the description of the mode for carrying out the invention and the drawings.

ADVANTAGE OF THE INVENTION According to this invention, the work which a worker performs can be monitored accurately, without preparing a large amount of data in advance.

1 is a diagram showing a schematic configuration of a work monitoring system; FIG. It is a figure which shows the hardware structural example of a work monitoring apparatus. It is a figure which shows the main functions with which a work monitoring apparatus is equipped. (a) is an example of image data, (b) is a drawing drawn by extracting the position of a marker shown in the image data and a determination target area, and (c) is a determination generated based on the image data. It is a result table. (a) to (c) are examples of image data and determination result patterns corresponding to the image data. 4 is a flowchart for explaining main processing; 9 is a flowchart for explaining marker color setting processing; 9 is a flowchart for explaining determination target area setting processing; 9 is a flowchart for explaining correspondence information registration processing; 10 is a flowchart for explaining determination result table generation processing; 9 is a flowchart for explaining marker position detection processing; 4 is a flowchart for explaining work monitoring processing; It is another example of the determination result table. It is a figure which shows the main functions with which the work monitoring apparatus of 2nd Embodiment is provided. 10 is a flowchart for explaining work monitoring processing according to the second embodiment; It is a figure which shows the main functions with which the work monitoring apparatus of 3rd Embodiment is provided. 10 is a flowchart for explaining work monitoring processing according to the third embodiment; It is a figure which shows the main functions with which the work monitoring apparatus of 4th Embodiment is provided. FIG. 14 is a flowchart for explaining work monitoring processing according to the fourth embodiment; FIG. It is a figure which shows the example of the period used as analysis object.

Embodiments will be described below with reference to the drawings. In the following description, the same reference numerals may be given to the same or similar configurations, and redundant description may be omitted. Further, in the following description, when it is necessary to distinguish the same type of configuration, an identifier (number, alphabet, etc.) may be written in parentheses after a symbol that generically refers to the configuration.

[First embodiment]
FIG. 1 shows a schematic configuration of a work monitoring system 1 described as a first embodiment. As shown in the figure, the work monitoring system 1 includes an image acquisition device 3, various sensors 4, and a work monitoring device 100 which is an information processing device. The image acquisition device 3 , various sensors 4 , and work monitoring device 100 are communicably connected via wired or wireless communication means 5 . The configuration of the communication means 5 is not necessarily limited. For example, a dedicated line.

The image acquisition device 3 (photographing device) is a device that acquires image data showing the worker 2 and the surroundings of the worker 2. For example, the image acquisition device 3 is a camera (digital camera) that acquires (photographs) image data of moving images and still images (RGB camera), infrared camera, thermography camera, Time Of Flight (TOF) camera, stereo camera, etc.).

Various sensors 4 are provided in the work environment where the worker 2 works, and output physical information about the worker 2 and the work environment. Various sensors 4 include, for example, a moving body detection sensor, a human sensor, a temperature sensor, a humidity sensor, an acceleration sensor, a speed sensor, an acoustic sensor (microphone), an ultrasonic sensor, a vibration sensor, a millimeter wave radar, a laser radar (LIDAR: Laser Imaging Detection and Ranging), an infrared depth sensor.

The work monitoring device 100 performs processing related to monitoring work of the worker 2 based on image data acquired by the image acquisition device 3 .

FIG. 2 shows a hardware configuration example of the work monitoring device 100. As shown in FIG. The work monitoring device 100 is an information processing device (computer) and includes a processor 11 , a main storage device 12 , an auxiliary storage device 13 , an input device 14 , an output device 15 and a communication device 16 .

The processor 11 is, for example, a device that performs arithmetic processing, and is a CPU (Central Processing
Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), AI (Artificial Intelligence) chip, and the like. The main storage device 12 is a device for storing programs and data, and includes, for example, ROM (Read Only Memory) (SRAM (Static Random Access Memory), NVRAM (Non Volatile RAM), Mask ROM (Mask Read Only Memory), PROM (Programmable ROM), etc.), RAM (Random Access Memory) (DRAM (Dynamic Random Access Memory), etc.), and the like. The auxiliary storage device 13 includes a hard disk drive, a flash memory, an SSD (Solid State Drive), an optical storage device (CD (Compact Disc), DVD (Digital Versatile
Disc), etc.). Programs and data stored in the auxiliary storage device 13 are read into the main storage device 12 as needed.

The input device 14 is a user interface that receives information from a user, such as a keyboard, mouse, card reader, and touch panel. The output device 15 is a user interface that outputs various types of information (display output, audio output, print output, etc.). These include an output device (speaker), a printing device, and the like.

The communication device 16 is a communication interface that communicates with other devices via the communication means 5, and includes, for example, a NIC (Network Interface Card), a wireless communication module, a USB (Universal Serial Interface) module, a serial communication module, and the like. Communication device 16 may also function as an input device to receive information from other devices with which it is communicatively connected. The communication device 16 can also function as an output device that transmits information to other devices with which it is communicatively connected. The work monitoring device 100 communicates with the image acquisition device 3 and various sensors 4 via the communication means 5 by the communication device 16 .

Various functions of the work monitoring device 100 are realized by the processor 11 reading and executing a program stored in the main storage device 12, or by hardware (FPGA, ASIC , AI chip, etc.).

FIG. 3 shows main functions of the work monitoring device 100. As shown in FIG. As shown in the figure, the work monitoring device 100 includes a storage unit 110 , a data acquisition unit 120 , a preparation setting processing unit 130 , a determination result table generation unit 140 and a monitoring processing unit 150 . In addition to the functions described above, the work monitoring apparatus 100 may further include functions such as an operating system, a device driver, a file system, and a DBMS (DataBase Management System).

Among the functions described above, the storage unit 110 stores image data 111 , sensor data 112 , marker color data 113 , determination target area data 114 , determination result table 115 , and determination result pattern/work content correspondence information 116 . The storage unit 110 stores such information (data) as, for example, a table of a database provided by a DBMS or a file provided by a file system.

The image data 111 is data acquired from the image acquisition device 3, and is, for example, frame data that is transmitted from the image acquisition device 3 and constitutes still image data or moving image data. The sensor data 112 is data obtained from various sensors 4 . Details of the marker color data 113, the determination target area data 114, the determination result table 115, and the determination result pattern/work content correspondence information 116 will be described later.

The data acquisition unit 120 acquires (receives) data sent from another device. As shown in the figure, the data acquisition section 120 includes an image data acquisition section 121 and a sensor data acquisition section 122 . Among these, the image data acquisition unit 121 acquires the image data 111 from the image acquisition device 3 . The sensor data acquisition unit 122 acquires sensor data 112 from various sensors 4 .

The preparation setting processing unit 130 performs processing related to setting of various information used when the work monitoring device 100 monitors the work of the worker 2 . As shown in the figure, the preparation setting processing section 130 includes a marker color setting processing section 131 , a determination target area setting processing section 132 , and a correspondence information setting processing section 133 .

Among these, the marker color setting processing unit 131 is based on the image data 111, in order to recognize the positions of the worker 2 and objects, the display provided on the body of the worker 2, the work target, the tools used for work, etc. (hereinafter referred to as "marker"). For example, the marker color setting processing unit 131 acquires color data (for example, data represented by RGB values) of an area in which a marker is shown in the image data 111, and an identifier of the marker (hereinafter referred to as a “marker ID”). .) and the color data is generated.

The determination target region setting processing unit 132 performs processing related to setting of a region for the image data 111 (hereinafter referred to as “determination target region”). The determination target area setting processing unit 132 receives, for example, the setting of the determination target area from the user. Further, the determination target area setting processing unit 132 automatically sets the determination target area in the image data. The determination target area setting processing unit 132 generates determination target area data 114 that is data including information specifying the set determination target area. The determination target area data 114 includes, for example, information that associates an identifier of the determination target area (hereinafter referred to as "determination target area ID") with information indicating the area in the coordinate system of the image data.

The correspondence information setting processing unit 133 determines that objects with markers (predetermined parts of the body of the worker 2, work objects, objects related to work such as tools used by the worker 2) exist in each determination target region ( The pattern of the judgment result table 115 (hereinafter referred to as "judgment result pattern"), which is data including information indicating whether or not the (referred to as "work content information"), and processing related to the setting of the determination result pattern/work content correspondence information 116, which is information associated with. The work content information includes information indicating whether the work of the determination result pattern is normal work or abnormal work (hereinafter referred to as "normal/abnormal division"). . Details of the determination result table 115 will be described later.

The determination result table generator 140 generates the determination result table 115 based on the image data 111 . As shown in the figure, the determination result table generation unit 140 includes a marker position detection processing unit 141 and a determination result table generation processing unit 142 .

Among them, the marker position detection processing unit 141 uses the marker color data 113 to detect the position where the marker appears in the image data 111 (in this example, the position of the center of gravity of the area where the marker moves).

The determination result table generation processing unit 142 generates the determination result table 115 by comparing the positions of the detected markers and the determination target area data 114 .

An example of the image data 111 is shown in FIG. The illustrated image data 111 includes a plurality of markers 42(1) to 42(4) provided on objects 41(1) to 41(4), respectively.
) is shown. Determination target areas 43(1) to 43(3) are also set.

FIG. 4(b) is drawn by extracting the positions of the markers 42(1)-(4) and the determination target regions 43(1)-(3) appearing in the image data 111 shown in FIG. 4(a). It is a diagram. Although the determination target areas 43(1) to 43(3) are rectangular frames in this example, they may be of other shapes.

FIG. 4(c) is a judgment result table 115 generated based on the image data 111 shown in FIG. 4(a). As shown in the figure, the determination result table 115 is a two-dimensional table in which marker IDs are set in the row direction and determination target area IDs are set in the column direction. Further, as shown in the figure, an item (hereinafter referred to as "Visible") indicating whether the marker is visible or invisible (whether or not it is reflected in the image data 111) is provided in the column direction. Note that the determination result pattern described above also has the same configuration as the determination result table 115 .

In the case of this example, for example, for an object with a marker ID of "Object 1", since it appears in the image data 111, the value of "Visible" is set to "True", and the determination target area ID is "Area 1". Since it is inside the area, the item is set to "True", and since it is outside the judgment target area ID "Area 2", the item is set to "False", and the judgment target area ID is "Area 3". Since it is outside the area, the item is set to "False".

As for the object whose marker ID is "Object 2", the value of "Visible" is set to "True" because it is reflected in the image data 111, and it is within the determination target area whose determination target area ID is "Area 1". Therefore, the item is set to "True", and since it is in the determination target area with the determination target area ID of "Area 2", the item is set to "True" and the determination target with the determination target area ID of "Area 3" Since it is outside the area, the item is set to "False".

As for the object with the marker ID "Object 3", the value of "Visible" is set to "True" because it is reflected in the image data 111, and it is outside the determination target area with the determination target area ID "Area 1". Therefore, the item is set to "False" and the judgment target area ID is "Area 2"
This item is set to "False" because it is outside the determination target area of "Area 3", and the item is set to "True" because it is within the determination target area whose determination target area ID is "Area 3".

As for the object whose marker ID is "Object 4", the value of "Visible" is set to "True" because it appears in the image data 111, and it is outside the determination target area whose determination target area ID is "Area 1". Therefore, the item is set to "False" and the judgment target area ID is "Area 2"
, the item is set to "False" because it is outside the determination target area of "Area 3".

For the object with the marker ID "Object 5", since it is not shown in the image data 111, the value of "Visible" is set to "False", and the determination target area IDs are "Area 1", "Area 2", The value of each item of "Area 3" is set to "False".

The content of the determination result table 115 shown above represents the characteristics of each work performed by the worker 2 . Therefore, by preparing in advance a judgment result table 115 based on the image data 111 acquired for each work (normal work or abnormal work) performed by the worker 2 as a judgment result pattern, the worker 2 can newly The work performed by the worker 2 can be specified easily and accurately based on the image data 111 showing the work.

FIG. 5 shows examples of judgment result patterns (judgment result table 115) generated for individual works performed by the worker 2 (three examples indicated by (a) to (c) in FIG. 5). Note that the determination result table 115 has an item of "Visible" and includes information as to whether or not the marker is shown in the image data 111, so the work of the worker 2 can be specified with high accuracy. . Further, since the positions of the markers appearing in the image data 111 can be automatically specified by using the information processing device, the user simply sets the determination target area in the image data 111, and the determination result table can be easily obtained. 115 can be generated. As described above, according to the work monitoring device 100 of the present embodiment, it is possible to easily realize a mechanism for monitoring the work of the worker 2 at individual sites where different works are performed.

Returning to FIG. 3 , the monitoring processing unit 150 monitors the work of the worker 2 based on the image data 111 sent from the image acquisition device 3 . As shown in the figure, the monitoring processing unit 150 includes a determination result table acquisition unit 151 and an abnormality presence/absence determination processing unit 152 .

Of these, the determination result table acquisition unit 151 acquires the determination result table 115 generated based on the image data 111 from the determination result table generation unit 140 .

The abnormality presence/absence determination processing unit 152 determines whether the work of the worker 2 is normal or abnormal based on the determination result table 115, and outputs the determination result. For example, the abnormality presence/absence determination processing unit 152 determines that the determination result table 115 generated based on the image data 111 has the normal/abnormal classification registered in the determination result pattern/work content correspondence information 116 set to "normal". If it matches with any of the determination result patterns, it is determined that the work of the worker 2 is normal, and if it does not match any pattern, it is determined that the work of the worker 2 is abnormal. Further, the abnormality presence/absence determination processing unit 152, for example, determines whether the determination result table 115 generated based on the image data 111 is a determination result pattern whose normal/abnormal classification is "abnormal" registered in the determination result pattern/work content correspondence information 116. If it matches any one of them, it is determined that the work of the worker 2 is abnormal, and if it does not match any of them, it is determined that the work of the worker 2 is normal. Note that matching between the determination result table 115 and the determination result pattern does not necessarily mean complete matching, and for example, matching may be determined if the degree of mismatch is within a preset allowable range.

<Description of processing>
Next, processing performed by the work monitoring device 100 will be described. In the following description, it is assumed that while the work monitoring device 100 is monitoring the work of the worker 2, the photographing range of the image acquisition device 3 is fixed. It is also assumed that each marker has a different color.

FIG. 6 is a flowchart for explaining the processing (hereinafter referred to as "main processing S600") performed by the work monitoring device 100 when monitoring the work of the worker 2 on site.

As shown in the figure, the preparation setting processing unit 130 first performs processing for setting information necessary for monitoring the work of the worker 2 (marker color setting processing S611, determination target area setting processing S612, and correspondence information registration). Processing S613) is performed. Details of each of these processes will be described later.

Subsequently, the monitoring processing unit 150 performs processing for monitoring the work of the worker 2 (work monitoring processing S614).

FIG. 7 is a flowchart for explaining the details of the marker color setting process S611 in FIG.

As shown in the figure, first, the image data acquisition unit 121 acquires the image data 111 from the image acquisition device 3 (S711).

Subsequently, the marker color setting processing unit 131 acquires the color data (RGB data) of the marker area shown in the image data 111 (S712). When a plurality of markers appear in the image data 111, the marker color setting processing unit 131 acquires color data of each marker area.

Subsequently, the marker color setting processing unit 131 converts the acquired color data into data in a predetermined color space (HSV (Hue Saturation Value), HSB (Hue Saturation Brightness), etc.) (S713). Note that the conversion to the color space may not necessarily be performed, and whether or not to perform the conversion may be determined according to the required monitoring accuracy (accuracy in determining whether the work of the worker 2 is normal or abnormal), for example. Just do it. The same applies to which color space is selected.

Subsequently, the marker color setting processing unit 131 stores the converted data as the marker color data 113 in association with the marker ID (S714).

FIG. 8 is a flowchart for explaining the details of the determination target area setting process S612 in FIG.

As shown in the figure, first, the image data acquisition unit 121 acquires the image data 111 from the image acquisition device 3 (S811).

Subsequently, the determination target area setting unit 132 sets one or more determination target areas in the acquired image data 111 (S812). The determination target area setting unit 132 sets the determination target area through interactive processing with the user, for example. It should be noted that the user appropriately sets the determination target area so as to ensure the required monitoring accuracy based on, for example, the relationship between the setting content of the determination target area in the past and the determination accuracy. Further, the determination target area setting processing unit 132 automatically sets the determination target area so as to ensure the required monitoring accuracy, based on the relationship between the setting contents of the determination target area in the past and the determination accuracy, for example.

Subsequently, the determination target area setting unit 132 stores the set determination target area as the determination target area data 114 (S813).

FIG. 9 is a flowchart for explaining the details of the correspondence information registration process S613 of FIG.

As shown in the figure, first, the image data acquisition unit 121 acquires the image data 111 from the image acquisition device 3 (S911).

Subsequently, the correspondence information setting processor 133 sends the acquired image data 111 to the determination result table generator 140 . The determination result table generation unit 140 performs a process of acquiring the determination result table 115 based on the sent image data 111 (hereinafter referred to as "determination result table generation process S912"), and generates the generated determination result table 115. Return to the correspondence information setting processing unit 133 . Details of the determination result table generation processing S912 will be described later.

Subsequently, the correspondence information setting processing unit 133 receives input (setting) of work content information for the image data 111 from the user (S913).

Subsequently, the correspondence information setting processing unit 133 associates the determination result table 115 sent from the determination result table generation unit 140 with the received work content information and registers them in the determination result pattern/work content correspondence information 154 ( S914).

FIG. 10 is a flowchart for explaining the details of the determination result table generation processing S912 of FIG.

As shown in the figure, first, the marker position detection processing unit 141 performs processing for detecting the position of each marker included in the image data 111 (hereinafter referred to as "marker position detection processing S1011"). Details of the marker position detection processing S1011 will be described later.

Subsequently, the determination result table generation processing unit 142 acquires the determination target region data 114 (S1012).

Subsequently, the determination result table generation processing unit 142 compares the detected marker positions with the acquired determination target region data 114 to generate the determination result table 115 (S1013).

FIG. 11 is a flowchart for explaining details of the marker position detection processing S811 of FIG.

As shown in the figure, first, the marker position detection processing unit 141 acquires the marker color data 113 (S1111). Incidentally, when the marker color data 113 of a plurality of markers exist, the marker position detection processing section 141 acquires all the marker color data 113 .

Subsequently, the marker position detection processing unit 141 extracts an area corresponding to the marker color data 113 from the image data 111 (an area whose degree of coincidence or similarity is equal to or less than a preset threshold; hereinafter referred to as a "marker area"). is detected (S1112). Note that when the marker color data 113 of a plurality of markers are acquired, the marker area is detected for each marker color data.

Subsequently, the marker position detection processing unit 141 removes noise information from the detected marker area using, for example, known noise removal techniques (image processing, morphology calculation, etc.) (S1113).

Subsequently, the marker position detection processing unit 141 identifies the center of gravity position (coordinates of the center of gravity) of the marker area (S1114).

Subsequently, the marker position detection processing unit 141 checks whether or not there are a plurality of marker areas corresponding to the same marker color data, and if there are a plurality of them, selects one of them (S1115). For example, the marker position detection processing unit 141 selects a marker area whose color is closest to the marker color data. Also, for example, the marker position detection processing unit 141 selects a marker area based on whether the position of the marker area in the image data 111 is within a predetermined range.

Subsequently, the marker position detection processing unit 141 sets the center of gravity of the selected marker area as the marker position (S1116).

FIG. 12 is a flowchart for explaining the details of the work monitoring process S614 of FIG.

As shown in the figure, first, the image data acquisition unit 121 acquires one or more pieces of image data 111 from the image acquisition device 3 (S1211). This image data 111 is, for example, a one-frame image of moving image data showing the state of the worker 2 .

Subsequently, the determination result table acquisition unit 151 passes the acquired image data 111 to the determination result table generation unit 140, and acquires the determination result table 115 from the determination result table generation unit 140 (S1212). Note that the processing of S1212 is the same as the determination result table generation processing S912 described above, so the description thereof will be omitted.

Subsequently, the abnormality presence/absence determination processing unit 152 determines whether or not the work of the worker 2 is normal based on the obtained determination result table 115 (S1213), and outputs the determination result to the output device 15 (S1
214).

Subsequently, the monitoring processing unit 150 determines whether or not the work of the worker 2 has ended (S1215). If the work has been completed (S1215: YES), the monitoring process S414 ends, and if the work has not been completed (S1215: NO), the process returns to S1211. Note that the monitoring processing unit 150 determines whether or not the work is finished based on information input by the user, for example.

As described above in detail, according to the work monitoring device 100 of the present embodiment, by preparing the determination result pattern in advance, the work can be detected from the image data 111 newly showing the work of the worker 2. It is possible to accurately determine whether the work performed by the person 2 is normal or abnormal. Therefore, it is possible to easily realize a mechanism for monitoring the work of the worker 2 at each site.

By the way, the determination result table 115 shown above includes information on markers (information on whether or not the position of the marker is within the determination target area and information on whether or not the marker appears in the image data 111). However, the determination result table 115 and the determination result pattern further include sensor data information acquired from the various sensors 4 when the image data 111 is acquired, and the monitoring processing unit 150 also considers the sensor data values. The determination result table 115 and the determination result pattern/work content correspondence information 116 may be compared with each other to identify the work and determine whether or not there is an abnormality in the work. By further including sensor data information in the determination result table 115 and the determination result pattern, it is possible to express the characteristics of each work in more detail, and it is possible to improve the determination accuracy of the work.

FIG. 13 shows an example of the determination result table 115 in that case. The determination result table 115 illustrated includes, in addition to each item of the determination result table 115 described above, items ("Sensor 1", " Sensor 2”). It also includes items (“Value 1” and “Value 2”) for which sensor data values are set in the column direction.

[Second embodiment]
The work monitoring device 100 of the second embodiment uses the mechanism of monitoring the work of the worker 2 in the first embodiment to monitor the time required for the work of the worker 2 (hereinafter referred to as "work time"). The work performed by the worker 2 is monitored (determined whether the work is normal or abnormal) by comparing the acquired work time with a preset standard work time. The following description will focus on portions that differ from the first embodiment.

FIG. 14 shows main functions of the work monitoring device 100 of the second embodiment. As shown in the figure, the storage unit 110 of the work monitoring device 100 of the second embodiment stores information for each work performed by the worker 2 in addition to the information stored in the storage unit 110 of the work monitoring device 100 of the first embodiment. A standard work time per work 117, which is data including information indicating the standard work time of each work, is stored. Further, the abnormality presence/absence determination processing unit 152 of the work monitoring device 100 of the second embodiment has a function of determining whether the work performed by the worker 2 is normal or abnormal by comparing the standard work time and the work time.

FIG. 15 is a flowchart for explaining work monitoring processing S614 in the second embodiment. Other processes in the second embodiment (other processes described in the first embodiment performed by the work monitoring device 100) are the same as those in the first embodiment, so descriptions thereof will be omitted.

As shown in the figure, first, the monitoring processing unit 150 initializes "work time" and "previous work", which are work variables used in the following processes (S1511). In addition, the accumulated work time when the worker 2 is performing a certain work is set in the "work time". Also, in the "previous work", S1
In the loop processing from 512, information indicating the previously specified work (for example, the work content information described above) is set.

Subsequently, the image data acquisition unit 121 acquires one or more pieces of image data 111 from the image acquisition device 3 (S1512). This image data 111 is a one-frame image of moving image data showing the state of the worker 2 . Note that in the loop processing of S1512 to S1517, the image data 111 of one frame of the moving image data are sequentially selected in time series.

Subsequently, the determination result table acquisition unit 151 passes the acquired image data 111 to the determination result table generation unit 140, and acquires the determination result table 115 from the determination result table generation unit 140 (S1513). Note that the processing of S1513 is the same as the determination result table generation processing S912 described in the first embodiment, so description thereof will be omitted.

Subsequently, the abnormality presence/absence determination processing unit 152 identifies the work performed by the worker 2 by comparing the acquired determination result table 115 with the determination result pattern/work content correspondence information 116 (S1514).

Subsequently, the abnormality presence/absence determination processing unit 152 determines whether or not the identified work is the same as the “previous work” (S1515). If the specified work is the same as the "previous work" (S1515: YES), the process proceeds to S1516. In the first loop, the process of S1515 is skipped and the process proceeds to S1516.

If the specified work is different from the "previous work" (S1515: NO), the process proceeds to S1518.

In S1516, the abnormality presence/absence determination processing unit 152 updates the "working time" (for example, adds the time from the previous update to the present time).

In S1517, the abnormality presence/absence determination processing unit 152 sets the work identified in S1514 (for example, work content information) as the "previous work". After that, the process returns to S1512.

In S1518, the abnormality presence/absence determination processing unit 152 acquires the standard work time of the work identified in S1514 from the determination result pattern/work content correspondence information 116, compares the "work time" with the standard work time, determines whether or not the work being performed is abnormal. For example, the abnormality presence/absence determination processing unit 152 determines that the work performed by the worker 2 is normal if the difference between the “work time” and the standard work time is equal to or less than a predetermined time, and the difference exceeds the predetermined time. If so, the work performed by the worker 2 is determined to be abnormal. Incidentally, by appropriately setting the predetermined time, the determination may be made while taking into consideration the time the worker 2 spends other than work such as break time.

In S<b>1519 , the abnormality presence/absence determination processing unit 152 outputs the determination result to the output device 15 .

As described above, according to the above mechanism, by using the mechanism for monitoring the work of the worker 2 in the first embodiment, the work performed by the worker 2 can be monitored based on the work time of the work performed by the worker 2. It is possible to realize a mechanism for monitoring whether is normal or abnormal.

[Third Embodiment]
The work monitoring device 100 of the third embodiment monitors whether or not the worker 2 is performing work according to the correct work order using the work monitoring mechanism of the worker 2 in the first embodiment. The following description will focus on portions that differ from the first embodiment.

FIG. 16 shows main functions of the work monitoring device 100 of the third embodiment. As shown in the figure, the storage unit 110 of the work monitoring device 100 of the third embodiment stores information indicating the correct work order in addition to the information stored by the storage unit 110 of the work monitoring device 100 of the first embodiment. It stores the work order information 118, which is the data included. Further, the abnormality presence/absence determination processing unit 152 of the work monitoring device 100 of the third embodiment has a function of determining whether or not the worker 2 is performing the work according to the regular work order set in advance.

FIG. 17 is a flowchart for explaining work monitoring processing S614 in the third embodiment. Other processes in the third embodiment (other processes described in the first embodiment performed by the work monitoring device 100) are the same as those in the first embodiment, so description thereof will be omitted.

As shown in the figure, the monitoring processing unit 150 first initializes the "previous work" which is a work variable used in the following processing (S1711). Information indicating the work specified last time in the loop processing from S1712 (for example, the above-mentioned work content information) is set in the "previous work".

Subsequently, the image data acquisition unit 121 acquires one or more pieces of image data 111 from the image acquisition device 3 (S1712). This image data 111 is a one-frame image of moving image data showing the state of the worker 2 . Note that in the loop processing of S1712 to S1716, the image data 111 of one frame of the moving image data are sequentially selected in chronological order.

Subsequently, the determination result table acquisition unit 151 passes the acquired image data 111 to the determination result table generation unit 140, and acquires the determination result table 115 from the determination result table generation unit 140 (S1713). Note that the processing of S1713 is the same as the determination result table generation processing S912 described in the first embodiment, so description thereof will be omitted.

Subsequently, the abnormality presence/absence determination processing unit 152 identifies the work performed by the worker 2 by comparing the acquired determination result table 115 with the determination result pattern/work content correspondence information 116 (S1714).

Subsequently, the abnormality presence/absence determination processing unit 152 compares the identified work, the “previous work”, and the work order information 118 to determine whether the identified work follows the work order in the work order information 118. (S1715). In the first loop, the process of S1715 is skipped and the process proceeds to S1716. If the specified work follows the work order in the work order information 118 (S1715: YES), the process proceeds to S1716. If the specified work does not follow the work order in the work order information 118 (S1715: NO), the process proceeds to S1717.

In S1716, the abnormality presence/absence determination processing unit 152 sets the work identified in S1714 (for example, work content information) as the "previous work". After that, the process returns to S1712.

In S<b>1717 , the abnormality presence/absence determination processing unit 152 outputs the determination result (information indicating that the work order does not follow the correct work order) to the output device 15 .

Thus, by using the work monitoring mechanism of the worker 2 in the first embodiment, it is possible to realize a mechanism for monitoring whether or not the work performed by the worker 2 follows the correct work order.

[Fourth embodiment]
The work monitoring apparatus 100 of the fourth embodiment uses the mechanism of monitoring the work of the worker 2 in the first embodiment to detect the period during which the work to be analyzed is being performed out of the period during which the worker 2 is performing. Identify. The following description will focus on portions that differ from the first embodiment.

FIG. 18 shows main functions of the work monitoring device 100 of the fourth embodiment. As shown in the figure, the storage unit 110 of the work monitoring device 100 of the fourth embodiment indicates the work to be analyzed in addition to the information stored by the storage unit 110 of the work monitoring device 100 of the first embodiment. It stores analysis target work information 119, which is data including information (for example, the work content information described above). Further, the abnormality presence/absence determination processing unit 152 of the work monitoring device 100 of the fourth embodiment has a function of specifying a period during which the work to be analyzed is being performed out of the period during which the worker 2 is performing.

FIG. 19 is a flowchart for explaining work monitoring processing S614 in the fourth embodiment. Other processes in the fourth embodiment (other processes described in the first embodiment performed by the work monitoring device 100) are the same as those in the first embodiment, so descriptions thereof will be omitted.

As shown in the figure, first, the image data acquisition unit 121 acquires one or more pieces of image data 111 from the image acquisition device 3 (S1911). This image data 111 is a one-frame image of moving image data showing the state of the worker 2 . Note that in the loop processing of S1911 to S1916, the image data 111 of one frame of the moving image data are sequentially selected in time series.

Subsequently, the determination result table acquisition unit 151 passes the acquired image data 111 to the determination result table generation unit 140, and acquires the determination result table 115 from the determination result table generation unit 140 (S1912). Note that the processing of S1912 is the same as the determination result table generation processing S912 described in the first embodiment, so description thereof will be omitted.

Subsequently, the abnormality presence/absence determination processing unit 152 identifies the work performed by the worker 2 by comparing the acquired determination result table 115 with the determination result pattern/work content correspondence information 116 (S1913).

Subsequently, the abnormality presence/absence determination processing unit 152 compares the identified work with the analysis target work information 119, and determines whether or not the identified work is the analysis target (S1914). If the identified work is to be analyzed (S1914: YES), the process proceeds to S1915. If the specified work is not the analysis target (S1914: NO), the process returns to S1911.

In S1915, the abnormality presence/absence determination processing unit 152 outputs to the output device 15 information indicating that the worker 2 is performing the work to be analyzed at the timing of the image data.

According to the above mechanism, when the worker 2 continuously performs a plurality of tasks, it is possible to output information indicating the period during which each task is performed. A person who intends to do so (hereinafter referred to as an "analyzer") can easily specify the period to be analyzed.

FIG. 20 shows an example of the period to be analyzed by the analyst. For example, as shown in (a), the analyst can use the above mechanism to specify the start and end points of the work to be analyzed. Further, for example, as shown in (b), the analyst can use the above mechanism to specify the start point of time to be analyzed, and specify a predetermined time from the start point as the period to be analyzed. Also, for example, as shown in (c), the analyst uses the above mechanism to specify the end point of the analysis target, and the analysis target period is from the point before the end point by a predetermined time to the end point. can be specified as

Although one embodiment of the present invention has been described in detail above, it goes without saying that the present invention is not limited to the above-described embodiment, and can be variously modified without departing from the scope of the invention. For example, the above embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations. Moreover, it is possible to add, delete, or replace a part of the configuration of the above embodiment with another configuration.

For example, the work monitoring process S614 described in each of the above embodiments may be performed on moving image data acquired in real time, or may be performed on recorded moving image data after the fact.

Further, each of the above configurations, functional units, processing units, processing means, and the like may be realized by hardware, for example, by designing a part or all of them using an integrated circuit. Moreover, each of the above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program for realizing each function. Information such as programs, tables, files, etc. that realize each function can be stored in memory, hard disks, recording devices such as SSDs (Solid State Drives), ICs, etc.
It can be placed on a recording medium such as a card, SD card, or DVD.

Further, in each of the above drawings, control lines and information lines are those considered necessary for explanation, and not all control lines and information lines for implementation are necessarily shown. For example, it may be considered that almost all structures are interconnected in practice.

Moreover, the arrangement form of various functional units, various processing units, and various databases of each information processing apparatus described above is merely an example. The arrangement form of various functional units, various processing units, and various databases can be changed to an optimum arrangement form from the viewpoint of the performance, processing efficiency, communication efficiency, etc. of hardware and software provided in these devices.

Also, the configuration of the database (schema, etc.) that stores the various data described above can be flexibly changed from the viewpoints of efficient use of resources, improvement of processing efficiency, improvement of access efficiency, improvement of search efficiency, and the like.

Further, instead of the position detection method using the markers described above, a sensor capable of automatically detecting the body of the worker 2 or an image recognition method may be used to detect the position of the object. Alternatively, the position of the object may be detected using a sensor capable of automatically detecting the work object or the tool used for the work, or using an image recognition technique. In this case, the automatically detected positional information of the body, work object, etc. is used to perform the same processing as in the above-described embodiment.

1 work monitoring system 2 worker 3 image acquisition device 4 various sensors 5 communication means 100 work monitoring device 111 image data 112 sensor data 113 marker color data 114 determination target area data 115 determination result table 116 determination result pattern/work content correspondence information 120 data acquisition unit 121 image data acquisition unit 122 sensor data acquisition unit 130 preparation setting processing unit 131 marker color setting processing unit 132 determination target area setting processing unit 133 correspondence information setting processing unit 140 determination result table generation unit 141 marker position detection processing unit 142 Judgment result table generation processing unit 150 Monitoring processing unit 151 Judgment result table acquisition unit 152 Abnormality presence/absence judgment processing unit S600 Main processing S611 Marker color setting processing S612 Judgment target area setting processing S613 Corresponding information registration processing S912 Judgment result table generation processing S1011 Marker position Detection processing S614 Work monitoring processing

Claims (13)

configured using an information processing device having a processor and memory,
an image data acquisition unit that acquires image data of a worker performing work;
an object position detection unit that acquires information indicating the positions of each of a plurality of objects appearing in the image data in the image data;
a determination result table containing information indicating results of determining whether or not the positions of the plurality of objects are included in each of one or more determination target areas set in the image data; a determination result table generation unit to generate;
a storage unit that stores determination result pattern/work content correspondence information that is information in which the determination result table generated based on the image data and work content information that is information indicating work in the image data are associated;
The determination result table is generated for the newly acquired image data, and the work performed by the worker is specified by comparing the generated determination result table with the determination result pattern/work content correspondence information. a monitoring processing unit;
with
The determination result table further includes information indicating whether the position of the object is included in the image data,
Work monitoring device. The work monitoring device according to claim 1,
Further comprising a sensor data acquisition unit that acquires sensor data, which is data output by a sensor provided at the site where the worker works,
The determination result table further includes the sensor data obtained from the sensor,
The monitoring processing unit generates the determination result table based on the newly acquired image data and the sensor data acquired from the sensor when the image data is acquired, and generates the determination result table and the determination result. identify the work that the person is doing by comparing with the pattern/work content correspondence information;
Work monitoring device. The work monitoring device according to claim 1 or 2 ,
The storage unit stores information indicating a standard work time required for the predetermined work,
The image data acquisition unit acquires time-series image data showing the worker working,
The determination result table generation unit generates the determination result table for each of the time-series image data,
The monitoring processing unit compares each of the determination result tables with the determination result pattern/work content correspondence information to specify the work performed by the worker, thereby enabling the worker to perform the predetermined work. Obtaining the required work time and monitoring the work performed by the worker by comparing the work time with the standard work time;
Work monitoring device. The work monitoring device according to claim 1 or 2 ,
The storage unit stores information indicating a standard work order for the plurality of work performed by the worker,
The image data acquisition unit acquires time-series image data showing the worker working,
The determination result table generation unit generates the determination result table for each of the time-series image data,
The monitoring processing unit compares each of the determination result tables with the determination result pattern/work content correspondence information to specify the work performed by the worker, thereby determining the work performed by the worker. Identifying an order and monitoring the work performed by the worker by comparing the identified work order with the standard work order;
Work monitoring device. The work monitoring device according to claim 1 or 2 ,
The storage unit stores information indicating a predetermined work to be performed by the worker,
The image data acquisition unit acquires time-series image data showing the worker working,
The determination result table generation unit generates the determination result table for each of the time-series image data,
The monitoring processing unit compares each of the determination result tables with the determination result pattern/work content correspondence information to specify the work being performed by the worker, thereby determining whether the predetermined work is to be started or finished. generate information indicating timing,
Work monitoring device. The work monitoring device according to claim 1,
further comprising a determination target area setting unit that receives setting of the determination target area for the image data;
Work monitoring device. The work monitoring device according to claim 1,
The object is at least one of a part of the worker's body, a work target, and a tool used by the worker during work,
Work monitoring device. The work monitoring device according to claim 1,
The object position detection unit acquires information indicating the position of the object in the image data by image recognition of a marker provided on the object.
Work monitoring device. An information processing device having a processor and memory,
Acquire image data showing the state of the worker working,
Acquiring information indicating respective positions in the image data for a plurality of objects appearing in the image data;
a determination result table containing information indicating results of determining whether or not the positions of the plurality of objects are included in each of one or more determination target areas set in the image data; generate and
storing determination result pattern/work content correspondence information that is information in which the determination result table generated based on the image data and work content information that is information indicating work in the image data are associated;
The determination result table is generated for the newly acquired image data, and the work performed by the worker is specified by comparing the generated determination result table with the determination result pattern/work content correspondence information. ,
The determination result table further includes information indicating whether the position of the object is included in the image data,
Work monitoring method. The work monitoring method according to claim 9 ,
The information processing device acquires sensor data, which is data output by a sensor provided at a site where the worker works,
The determination result table further includes the sensor data obtained from the sensor,
The information processing device generates the determination result table based on the newly acquired image data and the sensor data acquired from the sensor when the image data is acquired, and generates the determination result table and the determination result. identify the work that the person is doing by comparing with the pattern/work content correspondence information;
Work monitoring method. The work monitoring method according to claim 9 or 10 ,
The information processing device
storing information indicating a standard work time required for the predetermined work;
Acquiring time-series image data showing how the worker is working,
generating the determination result table for each of the time-series image data;
By comparing each of the determination result tables with the determination result pattern/work content correspondence information to specify the work performed by the worker, the work time required for the predetermined work by the worker is acquired. and monitoring the work performed by the worker by comparing the work time with the standard work time;
Work monitoring method. The work monitoring method according to claim 9 or 10 ,
The information processing device
storing information indicating a standard work order for the plurality of work performed by the worker;
Acquiring time-series image data showing how the worker is working,
generating the determination result table for each of the time-series image data;
By identifying the work performed by the worker by comparing each of the determination result tables with the determination result pattern/work content correspondence information, the order of the work performed by the worker is specified and specified. monitoring the work being performed by the worker by comparing the work order and the standard work order;
Work monitoring method. The work monitoring method according to any one of claims 9 and 10 ,
The information processing device
storing information indicating a predetermined work to be performed by the worker;
Acquiring time-series image data showing how the worker is working,
generating the determination result table for each of the time-series image data;
Information indicating the start or end timing of the predetermined work is generated by identifying the work performed by the worker by comparing each of the determination result tables with the determination result pattern/work content correspondence information. do,
Work monitoring method.

Images