JP2014182628A - Space importance determination device, space importance determination method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable a work surveyor who performs monitoring at a remote area to effectively confirm an image of a point needing confirmation for quality and safety management from the images capturing a visual field of a worker at a work site.SOLUTION: A space importance determination device comprises: an attention coordinate estimation section which estimates coordinates of a position to which an observer pays attention in an actual space as attention coordinates on the basis of a visual field image of the observer in the actual space and map information of the actual space; an area gazing degree calculation section which calculates an area gazing degree indicating how frequently a portion of an area in the actual space is gazed on the basis of the attention coordinates; and an area importance calculation section which calculates area importance indicating a degree of importance for a surveyor to confirm a certain area on the basis of the area gazing degree and work knowledge showing the degree of necessity to confirm for each area in accordance with a work content and a surrounding environment.

Description

  The present invention relates to a space importance level determination device, a space importance level determination method, and a program for supporting the monitoring work of a work supervisor.

A system for remotely performing quality control and safety management of work on a construction site is known.
For example, Patent Document 1 describes an witnessing work support system for supporting a witnessing work for monitoring the work status of a worker working at a work site from a remote location. This system is a network for communicating information necessary for witness work, and is installed at the work site. Image information during work at the work site taken by a predetermined device and work measured by the predetermined device. An information collection terminal that collects environmental information indicating environmental conditions at the site and transmits the collected image information and environmental information via the network, and image information and environmental information from the information collection terminal transmitted via the network And a center server that monitors at least the safety of the work site and the appropriateness of the work process based on the image information and the environment information.

JP 2006-285639 A

  However, the amount of image information during work at the work site is enormous, and includes an image of a part to be confirmed by the work supervisor for quality control and safety control, and an image that is not so important. In order to efficiently perform monitoring work, it is desirable to be able to preferentially confirm images of important places. However, the method described in Patent Document 1 cannot efficiently distinguish whether the video is related to an important part.

  In view of this, the present invention allows a work supervisor who performs monitoring at a remote location to efficiently capture a video of a part that needs to be confirmed for quality control or safety management from video images of the worker's field of view at the work site. One purpose is to be able to confirm.

  The space importance determination device according to the present invention uses, as coordinates of interest, coordinates of a position that the observer is paying attention to in the real space based on the visual field image of the observer in real space and the map information of the real space. An attention coordinate estimation unit to be estimated, an area gaze degree calculation unit that calculates an area gaze degree representing a frequency at which a part of the area in the real space was gaze based on the attention coordinates, and the area gaze degree An area importance calculation unit that calculates an area importance indicating how much a supervisor should confirm a certain area based on work knowledge indicating the degree of confirmation required for each area based on work content and surrounding environment, It is to be prepared.

  According to the present invention, a work supervisor who performs monitoring at a remote location can acquire an image of a location that needs to be confirmed for quality control or safety management.

The block diagram which shows the structure of the space importance determination apparatus by embodiment of this invention. The figure which shows the outline | summary of the remote monitoring system using the space importance determination apparatus by embodiment of this invention. The flowchart which shows operation | movement of the space importance determination apparatus by embodiment of this invention. The figure which shows an example of the estimation method of an attention coordinate in case map information is two-dimensional information by embodiment of this invention. The figure which shows an example of the estimation method of an attention coordinate in case map information is two-dimensional information by embodiment of this invention. The figure which shows an example of the setting method of the area in case map information is two-dimensional information by embodiment of this invention. The figure which shows the example of the display method of the area importance according to embodiment of this invention. The figure which shows the example of the display method of the area importance according to embodiment of this invention.

Next, embodiments for carrying out the present invention will be described in detail with reference to the drawings.
FIG. 1 is a block diagram showing a configuration of a spatial importance level judgment apparatus 10 according to an embodiment of the present invention. As shown in the figure, the spatial importance determination device 10 includes an attention coordinate estimation unit 101, an area gaze calculation unit 102, an area importance calculation unit 103, a map information storage unit 104, a business knowledge storage unit 105, a display device 120, An input device 130 is provided.

  The space importance level determination device 10 supports a monitor from a remote location to monitor the work status of a worker who works at a work site. FIG. 2 is a diagram showing an overview of a remote monitoring system using the space importance degree determination device 10. As shown in the figure, a worker 60 working at a work site is accompanied by a camera 61, a sensor 62 attached to the head, and a GPS (Global Positioning System) device 63.

A camera 61 attached to the operator's head or the like captures a visual field image of the operator. The captured image is transmitted to the space importance determination device 10 via the communication line 50.
The sensor 62 attached to the worker's head acquires data relating to the worker's viewpoint (viewing direction or viewpoint position on the visual field image). The sensor 62 may be, for example, a geomagnetic sensor, an acceleration sensor, a line-of-sight measurement device that combines a near-infrared image acquisition camera for an eyeball and a near-infrared light source, or a line-of-sight measurement device using a visible light image acquisition camera for an eyeball. . Data acquired by the sensor 62 is transmitted to the spatial importance determination device 10 via the communication line 50.

  The spatial importance determination device 10 can use the face direction (direction and elevation angle) estimated by the geomagnetic sensor or the acceleration sensor as the operator's line-of-sight direction. Alternatively, the space importance determination device 10 may use a value estimated by a combination of the face direction and the relative line-of-sight direction acquired by the line-of-sight measurement device as the worker's line-of-sight direction. . The line-of-sight direction relative to the face direction can be obtained by a method as described in Reference 1, for example. Specifically, the line-of-sight measurement device can measure the direction of the line of sight relative to the face direction from the shape of reflected light obtained by projecting near infrared rays onto the cornea. Alternatively, the line-of-sight measurement device can measure the line-of-sight direction relative to the face direction from the pupil or iris ellipse shape on the eyeball image obtained by photographing the eyeball with visible light.

The space importance degree determination device 10 may use the center point of the image captured by the camera 61 as the viewpoint position. Alternatively, the space importance determination device 10 obtains the plurality of specified coordinates on the image picked up by the camera 61 and the sensor 62 when the worker places the viewpoint on the place corresponding to the predetermined coordinates in the real space. The combination of the features of the eyeball image of the worker (the shape of the reflected light and the elliptical shape of the pupil or iris) is registered in advance. Then, the space importance determination device 10 compares the characteristics of the eyeball image of the worker acquired by the sensor 62 with the characteristics of the eyeball image registered in advance at the time of viewpoint measurement, and the eyeball having similar characteristics. The default coordinates that are paired with the image may be detected as the viewpoint position.
(Reference 1) Takehiko Ohno, “From eye-gaze interface to eye-gaze communication-Aiming for an environment with eye-gaze”, Information Processing Society of Japan Research Report HI, Human Interface Research Group Report 2001 (87), pp.171-178, 2001/09/13

  The GPS device 63 acquires worker position information. The positional information acquired by the GPS device 63 is transmitted to the spatial importance determination device 10 via the communication line 50.

  The spatial importance determination device 10 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and other memories, an external storage device for storing various information, an input interface, an output interface, and a communication interface. And a dedicated or general-purpose computer having a bus connecting them. The space importance determination device 10 may be configured by a single computer or may be configured by a plurality of computers connected to each other via a communication line.

  The attention coordinate estimation unit 101, the area gaze degree calculation unit 102, and the area importance degree calculation unit 103 correspond to modules of functions realized when the CPU executes a predetermined program stored in a ROM or the like. The map information storage unit 104 and the business knowledge storage unit 105 are implemented by an external storage device. The external storage device may be connected to the space importance degree determination device 10 via a network or the like.

The map information storage unit 104 stores two-dimensional or three-dimensional position information regarding an object existing in the real space.
The object existing in the real space is, for example, a three-dimensional object such as a ground surface before building a building, a building or a natural object around the site, or a building to be repaired. The two-dimensional position information related to the object may be, for example, information obtained by pairing the object in the two-dimensional construction drawing and its GPS information, or a still image obtained by photographing the object from the surroundings and the image thereof. It is good also as data which made the GPS information of a camera a pair. Further, the three-dimensional position information related to the object is, for example, each point on the surface of the object estimated based on the distance between the object acquired by the device such as a three-dimensional laser scanner and the GPS information of the device. It is good also as position data. The map information may be information created before construction. Moreover, map information may be updated whenever the shape and position of a target object change by construction.

  The business knowledge storage unit 105 stores information indicating the degree of confirmation required for each area (a part of the real space divided based on a preset standard) based on the process chart and construction type. ing.

The display device 120 is a display device such as a display, and displays various images in response to an image signal output from the CPU of the spatial importance determination device 10.
The display device 120 displays a field-of-view image of a worker who works on site. The visual field image of the worker is acquired via a camera worn by the worker, and transmitted to the space importance determination device 10 via a communication line.

  The input device 130 is various devices including a mouse, a keyboard, and the like, and is used when the user inputs various information to the spatial importance determination device 10.

Next, the operation of the space importance degree determination device 10 will be described.
FIG. 3 is a flowchart of the operation of the space importance degree determination device 10. First, the image data of the visual field image acquired by the worker's camera 61, the data regarding the worker's viewpoint acquired by the sensor 62, and the position information acquired by the GPS device 63 are used as the visual field image information to determine the spatial importance. It is transmitted to the attention coordinate estimation unit 101 of the device 10. Further, the map information stored in the map information storage unit 104 is input to the attention coordinate estimation unit 101 (step S1).

Next, the attention coordinate estimation unit 101 estimates the attention coordinates based on the input visual field image information (image data, viewpoint data, position information) and map information (step S2).
The attention coordinates are the coordinates of the position where the operator is paying attention at the work site. The attention coordinate estimation unit 101 sets the position coordinates in the real space of one or more representative points of the worker's visual field image as the attention coordinates. The representative points include, for example, the four corner points of the field image and the center point of the field image. The attention coordinate estimation unit 101 estimates and outputs position coordinates in the real space of representative points of the field image by mapping the field image to corresponding positions in the map information based on the field image information.

  4A and 4B are diagrams illustrating an example of a method of estimating a target coordinate when map information is two-dimensional information. As shown in FIG. 4A, the attention coordinate estimation unit 101 specifies the worker position P on the two-dimensional construction drawing based on the worker position information. Furthermore, the attention coordinate estimation unit 101 specifies the visual field direction vector Q starting from the position of the worker on the construction drawing and the visual field region R including a region that forms a predetermined angle α with respect to the visual direction vector Q. . Further, the attention coordinate estimation unit 101 identifies a region S (shaded rectangular region in FIG. 4A) included in the field-of-view region R among the objects (building A and building B). Next, the target coordinate estimation unit 101 sets the center of the region S as the target coordinate T as shown in FIG. 4B. Note that the coordinate of interest is not limited to the center of the region S, and may be another representative point.

  Note that the attention coordinate estimation unit 101 may compare a still image group obtained by photographing an object from the surroundings with a visual field image, and specify the vicinity of a portion moving to a similar still image as the region S.

  Similarly, when the map information is three-dimensional information, the attention coordinate estimation unit 101 applies the operator position, the three-dimensional gaze direction vector starting from the worker position, and the gaze direction vector. Thus, it is possible to specify a three-dimensional field of view that forms a predetermined angle α, and to specify a target coordinate from a region included in the three-dimensional field of view of the object surface.

  The attention coordinate estimation unit 101 outputs two-dimensional coordinates as attention coordinates when the map information is two-dimensional information. When the map information is three-dimensional information, the attention coordinate estimation unit 101 outputs the three-dimensional attention coordinates using the depth information of the visual field image.

  The attention coordinate estimation unit 101 may estimate the attention coordinates based on the viewpoint (attention point) of the worker acquired through the sensor 62 attached to the worker. Specifically, the attention coordinate estimation unit 101 estimates, as attention coordinates, a position where the operator has paid attention for a certain period of time. According to this method, the target coordinate can be estimated with higher accuracy.

  Next, the area gazing degree calculation unit 102 calculates an area gazing degree representing the frequency at which an area is being gazed based on the attention coordinates estimated in step S2 (step S3). The area is generated by dividing the map information stored in the map information storage unit 104 based on a preset criterion. That is, the area represents a part of the real space. The criteria for dividing the area are, for example, the area of a defined area such as a grid, the volume of a defined area such as a block, the volume and angle of an object, and the like.

  FIG. 5 is a diagram illustrating an example of an area setting method when the map information is two-dimensional information. In the example shown in FIG. 5, the area gaze degree calculation unit 102 divides the two-dimensional drawing into a grid shape, and divides the real space so that each rectangular area on the drawing corresponds to each partial space of the real space. To set the area. In addition, the area gaze degree calculation unit 102 classifies still image groups obtained by photographing the object from the surroundings based on the image similarity, so that the positions of the cameras that photograph the still images belonging to the same classification belong to the same partial space. The real space may be divided to set the area. In addition, when the map information is three-dimensional information, the area gaze degree calculation unit 102 performs an actual operation so that the point groups on the same plane among the point groups on the surface of the object belong to the same partial space. An area may be set by dividing a space.

  The area gazing degree calculation unit 102 counts the number of times attention coordinates in an area are detected within a predetermined time, the average time during which attention coordinates stay in the area within a predetermined time, The area gaze degree is calculated based on the number of people. For example, in the example of FIG. 5, the area gaze degree calculation unit 102 sets the highest area gaze degree for the area corresponding to the rectangular area A in which a large number of attention coordinates (stars) are detected.

  Further, the area gazing degree calculation unit 102 may calculate the number of workers paying attention within the area by adding a value weighted according to the attribute (skill level or knowledge) of the person. For example, the area gazing degree calculation unit 102 may weight the person who has a high skill level or a person who has qualifications regarding the construction content being watched.

  Next, the area importance calculation unit 103 calculates the area importance based on the area gazing degree calculated in step S3 and the business knowledge stored in the business knowledge storage unit 105 (step S4). The area importance is a value indicating how much the supervisor should confirm a certain area. The business knowledge is information indicating the degree of confirmation required for each area based on the work content and the surrounding environment, such as process schedule, construction type, and weather, and is a value independent of the area gaze degree. For example, as business knowledge, a certain area needs to be confirmed even if the area gaze degree is a predetermined value or less, a certain area needs to be confirmed regardless of the area gaze degree in the process A, but confirmation is not necessary in the process B. Information such as confirmation is necessary regardless of the area gaze degree in rainy weather, but confirmation is not necessary even if the area gaze degree is a predetermined value or more when the weather is good. For example, when the content of the business knowledge of a certain area is “requires confirmation even if the area gaze degree is equal to or less than a predetermined value”, the area importance degree calculation unit 103 calculates the area importance degree regardless of the calculated area gaze degree. Make it high.

  The area importance calculation unit 103 outputs the calculated area importance to the display device 120 (step S5). For example, as shown in FIG. 6, the area importance level calculation unit 103 may display the area importance levels in a list together with area position information. In addition, the area importance calculation unit 103 may display a list of position information of areas whose area importance is equal to or greater than a predetermined value.

  In addition, the area importance calculation unit 103 displays a virtual space video generated based on the visual field image of the worker and the map information on the display device 120 so as to clearly indicate an area having a high area importance on the virtual space video. It may be.

  For example, as shown in FIG. 7, the area importance calculation unit 103 groups the field-of-view image data for each area, selects representative images of each area at regular intervals, and displays them arranged in descending order of importance. You may do it. The area importance calculation unit 103 may select, as the representative image, an image with the least blurring in an image group obtained by capturing the same area, or sequentially select images of the same area captured by a plurality of workers. Alternatively, an image having a different angle or photographed object from the representative image presented in the past may be selected.

  In addition, the area importance calculation unit 103 generates an integrated image by integrating the field-of-view images grouped for each area, and displays the integrated images of each area arranged in descending order of importance at regular intervals. Good. The area importance calculation unit 103 may generate an integrated image using all images obtained by photographing the same area, or select only an image with less blurring from a group of images obtained by photographing the same area. May be generated. In addition, the area importance calculation unit 103 generates a new integrated image when a predetermined number or more of images that have changed from the integrated image presented in the past are collected from a group of images obtained by photographing the same area. Also good.

  Further, the area importance calculation unit 103 may display the representative image or the integrated image on the display device 120 with an effect of changing the visibility according to the area importance at regular intervals. For example, the area importance calculation unit 103 may emphasize a representative image or an integrated image of an area with high importance by highlighting or the like.

  Further, the area importance calculation unit 103 may add an effect (such as image reduction) that gradually reduces the visibility as time elapses after the representative image or the integrated image is updated. In addition, the supervisor may be able to select whether to display an image of a high importance area preferentially or to display an image of a low importance area priority.

  In this way, the area importance calculation unit 103 displays the image of each area on the display device 120 based on the area importance, so that the supervisor can instantly grasp the area to be confirmed with priority.

  As described above, according to the present embodiment, the supervisor confirms the area importance calculated based on the area gaze degree and the business knowledge, so that an area frequently attracted by many workers On the contrary, although no one is paying attention, it is possible to grasp an area to be confirmed (an area that is easily overlooked) and know where to focus on the work site. In addition, the supervisor often pays attention to many workers, but can grasp areas that are not necessarily important based on business knowledge, so that the area to be confirmed can be efficiently known.

A part or all of the above embodiment can be described as in the following supplementary notes, but is not limited thereto.
(Additional remark 1) The attention coordinate estimation part which estimates the coordinate of the position which the said observer is paying attention in the said real space as an attention coordinate based on the visual field image of the observer in the real space, and the map information of the said real space; ,
An area gaze degree calculating unit that calculates an area gaze degree representing a frequency at which a part of the area in the real space was gaze based on the attention coordinates;
Area importance level for calculating the area importance level indicating how much the supervisor should confirm a certain area based on the area attention level and the business knowledge indicating the necessary confirmation level for each area based on the work content and the surrounding environment A spatial importance determination device comprising: a calculation unit;

(Supplementary Note 2) The attention coordinate estimation unit
The spatial importance determination device according to appendix 1, wherein position coordinates of one or more representative points of the visual field image of the observer are estimated as attention coordinates.

(Supplementary Note 3) The area gaze degree calculating unit
At least the number of times the attention coordinate is detected in the area within a predetermined time, the average time during which the attention coordinate stays within the area within the predetermined time, and the number of observers who have focused on the area within the predetermined time The space importance degree determination device according to attachment 1, wherein the area gaze degree is calculated based on one.

(Supplementary Note 4) The area gaze degree calculating unit
The spatial importance determination device according to attachment 3, wherein the number of observers who have focused on the area is calculated by adding a value weighted according to the attribute of the person.

(Supplementary Note 5) The area importance calculation unit
The spatial importance determination device according to appendix 1, wherein the position information of each area and the area importance for each area are displayed as a list on a display device.

(Appendix 6) The area importance calculation unit
The space importance according to appendix 1, wherein a virtual space image generated based on the observer's visual field image and the map information is displayed on a display device, and the area having a high area importance is clearly indicated on the virtual space image. Degree determination device.

(Appendix 7) Estimating the coordinates of the position that the observer is paying attention to in the real space as the coordinates of interest based on the visual field image of the observer in the real space and the map information of the real space;
A step of calculating an area gaze degree representing a frequency at which a part of the area in the real space was gaze based on the attention coordinates;
A step of calculating an area importance level indicating how much a supervisor should confirm a certain area based on the area gaze degree and work knowledge indicating the degree of confirmation required for each area based on work content and surrounding environment; Spatial importance determination method with

(Appendix 8)
An attention coordinate estimation unit that estimates, as attention coordinates, coordinates of a position that the observer is paying attention to in the real space, based on a visual field image of the observer in real space, and map information of the real space;
An area gaze degree calculating unit that calculates an area gaze degree representing a frequency at which a part of the area in the real space was gaze based on the attention coordinates;
Area importance level for calculating the area importance level indicating how much the supervisor should confirm a certain area based on the area attention level and the business knowledge indicating the necessary confirmation level for each area based on the work content and the surrounding environment A program that functions as a calculation unit.

  DESCRIPTION OF SYMBOLS 10 Spatial importance determination apparatus, 50 Communication line, 60 Worker, 61 Camera, 62 Sensor, 63 GPS apparatus, 101 Attention coordinate estimation part, 102 Area gaze degree calculation part, 103 Area importance degree calculation part, 104 Map information storage part 105 business knowledge storage unit, 120 display device, 130 input device

Claims (8)

An attention coordinate estimation unit that estimates, as attention coordinates, coordinates of a position that the observer is paying attention to in the real space, based on a visual field image of the observer in real space, and map information of the real space;
An area gaze degree calculating unit that calculates an area gaze degree representing a frequency at which a part of the area in the real space was gaze based on the attention coordinates;
Area importance level for calculating the area importance level indicating how much the supervisor should confirm a certain area based on the area attention level and the business knowledge indicating the necessary confirmation level for each area based on the work content and the surrounding environment A spatial importance determination device comprising: a calculation unit; The attention coordinate estimation unit
The spatial importance determination apparatus according to claim 1, wherein position coordinates in real space of one or more representative points of the observer's visual field image are estimated as attention coordinates. The area gaze degree calculating unit
At least the number of times the attention coordinate is detected in the area within a predetermined time, the average time during which the attention coordinate stays within the area within the predetermined time, and the number of observers who have focused on the area within the predetermined time The spatial importance determination device according to claim 1, wherein the area gaze degree is calculated based on one. The area gaze degree calculating unit
The spatial importance determination apparatus according to claim 3, wherein the number of observers who have focused on the area is calculated by adding a value weighted according to the attribute of the person. The area importance calculation unit
The spatial importance determination device according to claim 1, wherein position information of each area and the area importance for each area are displayed as a list on a display device. The area importance calculation unit
2. The space according to claim 1, wherein a virtual space image generated based on the observer's visual field image and the map information is displayed on a display device, and the area having a high area importance is clearly indicated on the virtual space image. Importance determination device. Estimating coordinates of a position that the observer is paying attention to in the real space as attention coordinates based on the visual field image of the observer in real space and the map information of the real space;
A step of calculating an area gaze degree representing a frequency at which a part of the area in the real space was gaze based on the attention coordinates;
A step of calculating an area importance level indicating how much a supervisor should confirm a certain area based on the area gaze degree and work knowledge indicating the degree of confirmation required for each area based on work content and surrounding environment; Spatial importance determination method with Computer
An attention coordinate estimation unit that estimates, as attention coordinates, coordinates of a position that the observer is paying attention to in the real space, based on a visual field image of the observer in real space, and map information of the real space;
An area gaze degree calculating unit that calculates an area gaze degree representing a frequency at which a part of the area in the real space was gaze based on the attention coordinates;
Area importance level for calculating the area importance level indicating how much the supervisor should confirm a certain area based on the area attention level and the business knowledge indicating the necessary confirmation level for each area based on the work content and the surrounding environment A program that functions as a calculation unit.

Images