JP2014115700A - Confirmation requirement degree determination device, confirmation requirement degree determination method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable a work monitoring-person to efficiently confirm a video image related to important work among video images obtained by imaging the field of vision of a worker in the work site.SOLUTION: A device includes: a target detection part 101 for detecting an image relevant to a pre-registered target article from video images obtained by imaging a field of vision of a worker; a work item matching part 102 for determining a work item performed by the worker on the basis of the target article and a characteristic of an operation performed by the worker on the target article in the video image; an item content similarity calculation part 103 for comparing the characteristic of an operation performed by the worker on the target article in the video image with a characteristic of a correct operation of the work item pre-registered, to calculate similarity between the aforementioned operations; and a confirmation requirement degree calculation part 104 for calculating a confirmation requirement degree for the operation contained in the video image on the basis of the importance of the work item and the similarity.

Description

  The present invention relates to a confirmation degree determination apparatus, a confirmation degree determination method, and a program for supporting a monitoring task of a work supervisor.

A system for remotely performing quality control and safety management of work on a construction site is known.
For example, Patent Document 1 describes an witnessing work support system for supporting a witnessing work for monitoring the work status of a worker working at a work site from a remote location. This system is a network for communicating information necessary for witness work, and is installed at the work site. Image information during work at the work site taken by a predetermined device and work measured by the predetermined device. An information collection terminal that collects environmental information indicating environmental conditions at the site and transmits the collected image information and environmental information via the network, and image information and environmental information from the information collection terminal transmitted via the network And a center server that monitors at least the safety of the work site and the appropriateness of the work process based on the image information and the environment information.

JP 2006-285639 A

  However, the amount of image information during work at the work site is enormous, including images related to work to be confirmed by the supervisor, that is, important work and work that is prone to mistakes, and less important images. Is included. In order to perform the monitoring work efficiently, it is desirable to be able to preferentially check images of important work. However, the method described in Patent Document 1 cannot efficiently distinguish whether the video is related to important work.

  Accordingly, an object of the present invention is to enable a work supervisor to efficiently confirm a video related to an important work from videos obtained by imaging the field of view of the worker at the work site.

  The degree-of-confirmation determination device according to the present invention includes an object detection unit that detects an image corresponding to an object registered in advance from an image obtained by imaging a worker's field of view, the object, and the work A work item matching unit for determining a work item performed by the worker based on characteristics of an operation performed by the worker on the object in the video; and An item content similarity calculation unit that compares the characteristics of an operation performed on an object with the characteristics of a correct operation of the work item registered in advance and calculates the similarity of both operations; A degree-of-confirmation calculation unit that calculates the degree of necessity of the operation in the video based on the degree of importance and the similarity.

  ADVANTAGE OF THE INVENTION According to this invention, the work supervisor can confirm efficiently the image | video relevant to important work out of the image | video which imaged the visual field of the worker in a work site.

The block diagram which shows the structure of the confirmation required level determination apparatus by embodiment of this invention. The figure which shows the outline | summary of the remote monitoring system using the confirmation required level determination apparatus by embodiment of this invention. The flowchart which shows operation | movement of the confirmation required degree determination apparatus by embodiment of this invention.

Next, embodiments for carrying out the present invention will be described in detail with reference to the drawings.
FIG. 1 is a block diagram showing a configuration of a necessity confirmation device 10 according to an embodiment of the present invention. As shown in the figure, the confirmation level determination device 10 includes an object detection unit 101, a work item matching unit 102, an item content similarity calculation unit 103, a confirmation level calculation unit 104, an object data storage unit 105, and an operation content. A data storage unit 106, a display device 120, and an input device 130 are provided.

  The degree-of-confirmation determination device 10 assists the monitoring person to monitor the work situation of the worker who works at the work site from a remote place. FIG. 2 is a diagram showing an overview of a remote monitoring system using the confirmation degree determining device 10. As shown in the figure, the confirmation degree determination device 10 is connected to a camera 61 and a sensor 62 attached to a worker 60 working at a work site via a communication line 50. The worker's field-of-view image acquired by the camera 61 and the data regarding the movement of the worker's hand acquired by the sensor 62 are transmitted to the confirmation degree determination apparatus 10 via the communication line 50.

  The degree-of-confirmation determination device 10 applies a dedicated or general-purpose computer having a CPU, a memory such as a ROM or a RAM, an external storage device for storing various information, an input interface, an output interface, a communication interface, and a bus connecting them. be able to. The degree-of-confirmation determination apparatus 10 may be configured by a single computer or may be configured by a plurality of computers connected to each other via a communication line.

  The object detection unit 101, the work item matching unit 102, the item content similarity calculation unit 103, and the confirmation level calculation unit 104 have functions realized by the CPU executing a predetermined program stored in a ROM or the like. Corresponds to a module. The degree-of-confirmation calculation unit 104, the object data storage unit 105, and the work content data storage unit 106 are implemented by an external storage device. The external storage device may be connected to the confirmation degree determination device 10 via a network or the like.

  In the target object data storage unit 105, image data of a target object to be worked by the worker is registered. The object is, for example, a part such as a screw or a bolt or a work tool. The image data may include the coordinates of the position corresponding to each corner point and information on the position where the luminance changes greatly.

  In the work content data storage unit 106, information on the ideal operation content of the work and the importance of the work is registered for the work items of the worker for the object. The ideal operation includes information on the action items to be performed, the correct order of each action, the standard time taken for each action, the viewpoint movement when performing the action, and the hand movement when performing the action. . The importance of work is set higher for important work and work that is prone to mistakes.

The display device 120 is a display device such as a display, and displays various images in response to an image signal output from the CPU of the confirmation necessary degree determination device 10.
The display device 120 displays a field-of-view image of a worker who works on site. The worker's visual field image and motion image data are acquired via a camera worn by the worker, and transmitted to the confirmation degree determining apparatus 10 via a communication line.

  The input device 130 is a variety of devices including a mouse, a keyboard, and the like, and is used when a user inputs various types of information to the confirmation degree determination device 10.

Next, operation | movement of the confirmation degree determination apparatus 10 required is demonstrated.
FIG. 3 is a flowchart of the operation of the confirmation degree determining apparatus 10. First, a worker's visual field image is transmitted from the worker's camera and input to the object detection unit 101 of the necessity determination device 10 (step S1).

Next, the object detection unit 101 detects an image of the object registered in the object data storage unit 105 from the input field-of-view image (step S2).
The detection of the image of the object can be performed, for example, by template matching between each image data registered in the object data storage unit 105 and the visual field image.

  Next, the work item matching unit 102 determines a work item of an operation performed by the worker on the object detected in step S2 (step S3).

  In the work item matching unit 102, in addition to the visual field image transmitted from the worker's camera, information on hand movement (speed, angle, etc.) sent through the acceleration sensor worn by the worker on the arm. Is entered. The work item matching unit 102 compares the operation content information for each work item registered in the work content data storage unit 106 with the visual field image from the worker's camera and the hand movement information from the acceleration sensor. The work item currently performed by the worker is determined. Note that the number of matching work items is not limited to one, and a plurality of work items may be selected.

  Next, in the item content similarity calculation unit 103, the similarity between the work content that the worker actually performs on the object and the operation content of the work item registered in the work content data storage unit 106 is obtained. Calculate (step S4).

  In the item content similarity calculation unit 103, in addition to the field-of-view image transmitted from the worker's camera and the information on the hand movement transmitted from the acceleration sensor, the viewpoint of the worker obtained using infrared corneal reflection is stored. Movement information is input. The item content similarity calculation unit 103 is registered in the work content data storage unit 106 based on the visual field image of the worker, the information on the movement of the hand from the acceleration sensor, the information on the movement of the viewpoint of the worker, and the like. The ideal operation content of the work item specified in step S3 is compared with the work actually performed by the worker, and the similarity is calculated.

  Specifically, when each operation to be performed is not performed in the correct order, an operation not included in the operation to be performed is performed, and the time taken for each operation is different from the standard time, when performing an operation If it is determined that the viewpoint movement or hand movement is different, the similarity decreases according to the degree of each.

  If a plurality of work items are selected in step S3, the similarity is calculated for each work item.

  Next, the confirmation required degree calculation unit 104 confirms the necessity of the work performed by the worker based on the similarity calculated in step S4 and the importance of the work item registered in the work content data storage unit 106. The degree is calculated (step S5).

  The degree-of-confirmation calculation unit 104 increases the degree of confirmation as the importance of the work item is higher and the similarity of the operation is lower. Further, the degree of confirmation required calculation unit 104 may calculate the degree of confirmation required only when the similarity is equal to or greater than a predetermined value.

  In addition, when the similarity is calculated for a plurality of work items, the degree of confirmation required may be calculated only for the work item having the highest similarity.

  The degree-of-confirmation calculation unit 104 may output to the display device 120 a list that combines the degree of confirmation and information on the time at which the video of the corresponding operator's operation is recorded. The list may be displayed in descending order of necessity of confirmation.

Further, the degree-of-confirmation calculation unit 104 may extract an image of an operation in which the degree of confirmation is equal to or greater than a certain value and display it on the display device 120. The video is obtained by dividing the visual field image into predetermined sections. As a division method, the video may be divided at regular intervals, or the change point of the work may be detected from the change in the operation obtained from the view image and the change in the subject, and may be divided into work units.
Alternatively, the degree-of-confirmation calculation unit 104 may extract and display a representative still image from an operation video whose degree of confirmation is a certain value or more.

Next, a method for calculating the degree of confirmation will be described using a specific example.
An on-site worker wears a camera that captures a field-of-view image, an acceleration sensor that is worn on both hands, and an infrared line-of-sight detection device.

  An image of an operation in which the worker fastens the component 1 (object) to the component 2 with the bolt A and the bolt B is acquired through the camera and transmitted to the confirmation degree determination device 10. The object detection unit 101 of the confirmation degree determination device 10 detects the part 1 as an object.

  Next, the work item matching unit 102 determines that the work item being performed by the worker is “fasten component 1 to component 2” from the information on the movement of the hand by the visual field image and the acceleration sensor.

  Next, the item content similarity calculation unit 103 is an ideal operation pattern “work procedure (part 1 is bolted to the part 1) of the work item“ fasten part 1 to part 2 ”registered in the work content data storage unit 106. A and bolt B are stopped in the order of parts 2), the viewpoint is moved during the work and the hand moves (stopping while holding this position while looking at this position), working time (10 minutes) "and actually Are compared to calculate the degree of similarity. The degree of similarity is low when the order in which the bolts are stopped is different, the position of the viewpoint is different, or when the working time is much longer or shorter than 10 minutes.

  The degree-of-confirmation calculation unit 104 calculates the degree-of-confirmation by weighting the calculated similarity with the importance of the work item “fix component 1 to component 2”.

  As described above, according to the present embodiment, the supervisor can determine whether or not to confirm the video of the corresponding work based on the output degree of confirmation required. In addition, when there are a plurality of work images, it is possible to check with priority from the ones with a high degree of confirmation required, or determine that confirmation is not necessary for those with a degree of confirmation required that is a certain value or less.

  The ideal operation content of each work item registered in the work content data storage unit 106 may include an operation according to the surrounding environment and the construction plan in addition to the operation on the object. For example, when abnormal noise is generated during work, it is confirmed whether a specific cable is connected to a predetermined part, and when water is close, a material for preventing corrosion is installed.

  Also, from the movements acquired by the cameras and sensors attached to skilled workers, movement patterns that are not registered in the work content data storage unit 106 are extracted and stored, and these movement patterns are ideal movement patterns. May be added to the work content data storage unit 106.

  Further, the importance of the work item may be set to be high for an item for which the supervisor himself has performed a wrong action in the past. Or you may enable it to set highly about the item which many workers performed the wrong operation in the past. Further, the importance of the work having a high degree of relevance with the immediately preceding work may be increased.

  Also, the importance may be set based on the work contents of other workers detected at the same time. For example, when the worker A performs an operation of carrying an object with a crane, it is possible to increase the importance of the surrounding confirmation operation of the worker working around.

A part or all of the above embodiment can be described as in the following supplementary notes, but is not limited thereto.
(Supplementary Note 1) An object detection unit that detects an image corresponding to an object registered in advance from an image obtained by imaging the visual field of the worker;
A work item matching unit that determines a work item performed by the worker based on characteristics of the operation performed by the worker on the target in the video;
Item contents for calculating the similarity between both operations by comparing the characteristics of the operation performed by the worker on the object in the video with the characteristics of the correct operation of the work item registered in advance. A similarity calculator;
A degree of necessity determination device comprising: a degree of necessity calculation unit that calculates the degree of necessity of the operation in the video based on the importance of the work item and the similarity.

(Appendix 2) The object detection unit
The degree-of-confirmation degree determination apparatus according to appendix 1, wherein an image corresponding to the object is detected by comparing the image feature quantity of the object registered in advance with the image feature quantity in the video.

(Supplementary Note 3) The work item matching unit
The work item is determined by comparing a feature of hand movement registered in advance corresponding to the work item with a feature of hand movement of the worker in the video. Confirmation degree determination device.

(Supplementary Note 4) The item content similarity calculation unit includes:
The confirmation required according to any one of appendices 1 to 3, wherein the similarity is calculated based on at least one of a feature of the worker's hand movement, a feature of the worker's viewpoint movement, and an operation time. Degree determination device.

(Supplementary Note 5)
The degree of necessity determination device according to any one of supplementary notes 1 to 4, wherein the degree of necessity is increased as the degree of importance is higher and the degree of similarity is lower.

(Appendix 6) The work item matching unit
As work items being performed by the worker, a plurality of work items are selected as candidates,
The item content similarity calculation unit includes:
Calculate the similarity of each selected work item,
The confirmation required degree calculation unit
The degree-of-confirmation determination device according to any one of supplementary notes 1 to 5, wherein the degree of confirmation is calculated only for a work item having the highest degree of similarity.

(Appendix 7)
The degree of necessity determination device according to any one of appendices 1 to 6, wherein the degree of confirmation is calculated only for a work item whose similarity is a predetermined value or more.

(Supplementary Note 8) The degree of confirmation calculation required unit is
8. The degree-of-needs determination device according to any one of appendices 1 to 7, which outputs a list that combines the calculated degree of confirmation and the time when the corresponding operation video is recorded in the video.

(Supplementary Note 9) The confirmation degree calculation unit requires
The degree-of-confirmation determining device according to attachment 8, wherein the list is displayed in descending order of degree of confirmation.

(Supplementary Note 10) The confirmation required degree calculation unit
The degree-of-confirmation degree determination device according to any one of appendices 1 to 7, wherein an image of an operation in which the degree of confirmation is equal to or greater than a predetermined value is extracted.

(Supplementary Note 11) The required degree of calculation calculator
8. The degree-of-confirmation determination device according to any one of supplementary notes 1 to 7, wherein a representative still image is extracted and displayed from an operation video whose degree of confirmation is equal to or greater than a predetermined value.

(Additional remark 12) The process of detecting the image applicable to the target registered beforehand from the image | video which imaged the visual field of the worker,
Determining a work item performed by the worker based on characteristics of the object and an operation performed by the worker on the target in the video;
Comparing the characteristics of the operation performed by the worker on the object in the video with the characteristics of the correct operation of the work item registered in advance, and calculating the similarity between the two operations. ,
And a step of calculating a degree of necessity of the operation in the video based on the degree of importance of the work item and the degree of similarity.

(Supplementary note 13)
An object detection unit for detecting an image corresponding to an object registered in advance from an image obtained by imaging a worker's field of view;
A work item matching unit that determines a work item performed by the worker based on characteristics of the operation performed by the worker on the target in the video;
Item contents for calculating the similarity between both operations by comparing the characteristics of the operation performed by the worker on the object in the video with the characteristics of the correct operation of the work item registered in advance. A similarity calculator;
A program that functions as a necessary degree calculation unit that calculates the degree of necessity of the operation in the video based on the importance of the work item and the similarity.

  DESCRIPTION OF SYMBOLS 10 Confirmation required determination apparatus, 50 Communication line, 60 Worker, 61 Camera, 62 Sensor, 101 Object detection part, 102 Work item matching part, 103 Item content similarity calculation part, 104 Confirmation degree calculation part, 105 Object Object data storage unit, 106 Work content data storage unit, 120 Display device, 130 Input device

Claims (13)

An object detection unit for detecting an image corresponding to an object registered in advance from an image obtained by imaging a worker's field of view;
A work item matching unit that determines a work item performed by the worker based on characteristics of the operation performed by the worker on the target in the video;
Item contents for calculating the similarity between both operations by comparing the characteristics of the operation performed by the worker on the object in the video with the characteristics of the correct operation of the work item registered in advance. A similarity calculator;
A degree of necessity determination device comprising: a degree of necessity calculation unit that calculates the degree of necessity of the operation in the video based on the importance of the work item and the similarity. The object detection unit includes:
The degree-of-confirmation degree determination device according to claim 1, wherein an image corresponding to the object is detected by comparing an image feature amount of the object registered in advance with an image feature amount in the video. The work item matching unit includes:
The work item is determined by comparing a hand movement characteristic registered in advance corresponding to the work item with a hand movement characteristic of the worker in the video. Confirmation degree determination device. The item content similarity calculation unit includes:
4. The key according to claim 1, wherein the similarity is calculated based on at least one of a feature of the worker's hand movement, a feature of the worker's viewpoint movement, and an operation time. 5. Confirmation degree determination device. The confirmation required degree calculation unit
The degree of necessity determination device according to any one of claims 1 to 4, wherein the degree of necessity is increased as the degree of importance is higher and the degree of similarity is lower. The work item matching unit includes:
As work items being performed by the worker, a plurality of work items are selected as candidates,
The item content similarity calculation unit includes:
Calculate the similarity of each selected work item,
The confirmation required degree calculation unit
The degree of necessity determination device according to any one of claims 1 to 5, wherein the degree of confirmation is calculated only for a work item having the highest degree of similarity calculated. The confirmation required degree calculation unit
The degree of necessity determination device according to any one of claims 1 to 6, wherein the degree of confirmation is calculated only for a work item whose similarity is a predetermined value or more. The confirmation required degree calculation unit
8. The degree-of-confirmation determination device according to claim 1, wherein a list of the calculated degree-of-confirmation required and a time when the corresponding operation video is recorded in the video is output. The confirmation required degree calculation unit
The degree-of-confirmation determination apparatus according to claim 8, wherein the list is displayed in descending order of degree of confirmation. The confirmation required degree calculation unit
The degree-of-confirmation degree determination device according to any one of claims 1 to 7, wherein an image of an operation in which the degree of confirmation is equal to or greater than a predetermined value is extracted. The confirmation required degree calculation unit
The degree-of-confirmation degree determination apparatus according to claim 1, wherein a representative still image is extracted and displayed from a video of an operation in which the degree of confirmation is a predetermined value or more. A step of detecting an image corresponding to an object registered in advance from an image obtained by imaging a worker's field of view;
Determining a work item performed by the worker based on characteristics of the object and an operation performed by the worker on the target in the video;
Comparing the characteristics of the operation performed by the worker on the object in the video with the characteristics of the correct operation of the work item registered in advance, and calculating the similarity between the two operations. ,
And a step of calculating a degree of necessity of the operation in the video based on the degree of importance of the work item and the degree of similarity. Computer
An object detection unit for detecting an image corresponding to an object registered in advance from an image obtained by imaging a worker's field of view;
A work item matching unit that determines a work item performed by the worker based on characteristics of the operation performed by the worker on the target in the video;
Item contents for calculating the similarity between both operations by comparing the characteristics of the operation performed by the worker on the object in the video with the characteristics of the correct operation of the work item registered in advance. A similarity calculator;
A program that functions as a necessary degree calculation unit that calculates the degree of necessity of the operation in the video based on the importance of the work item and the similarity.

Images