WO2023276305A1

WO2023276305A1 - Surveillance assisting system

Info

Publication number: WO2023276305A1
Application number: PCT/JP2022/011334
Authority: WO
Inventors: 紗織平田
Original assignee: コニカミノルタ株式会社
Priority date: 2021-06-29
Filing date: 2022-03-14
Publication date: 2023-01-05

Abstract

This surveillance assisting system comprises: a surveillance-object coordinate estimation unit for estimating the spatial coordinates of a surveillance object included in a captured image, on the basis of a camera's spatial coordinates, azimuth, angle of elevation, and angle of view, which are entered at the time of image capture; and a surveillance-object distribution map generation unit that adds, to a map 81c and a side view 82c, icons i1, i2 representing the surveillance object, at locations based on the spatial coordinates of said surveillance object, and that outputs the same to a display device.

Description

monitoring support system

The present invention relates to a monitoring support system.

For example, in facilities such as production facilities that produce natural gas and oil, and production plants that use gas to produce chemical products, there are pipes and devices that allow gas to flow within the premises and buildings, and gas is stored. tank is placed. In these facilities, a monitoring support system that detects leaks of combustible or toxic gases at an early stage by imaging structures such as the tanks with an infrared camera that visualizes gases (for example, Patent Document 1). is provided. Particularly in a vast facility, the required imaging field of view is wide and spans multiple locations, so it is necessary to acquire images of a large number of imaging fields of view using a plurality of cameras or a plurality of times of imaging. Therefore, in the monitoring support system, in order to efficiently acquire the necessary images, it is necessary to indicate on the display in advance which point in the facility the image should be taken from and which azimuth angle (pan) the camera should be directed at. This is set by adding a fan-shaped imaging range extending from each imaging point to the displayed map of the facility (for example, Patent Document 1). In addition, using a camera equipped with an infrared camera and a general camera that captures visible light, visible images (color images) obtained at the same time when the leaked gas is included in the infrared images obtained A system has been disclosed that performs image processing for superimposing a figure representing a gas on the image (for example, Patent Documents 2 to 4).

In addition, it is desirable to be able to easily know at which point in the facility the large number of acquired image data were captured. Therefore, when an image is taken with a digital camera equipped with a GPS (Global Positioning System) and a direction sensor, the position data and direction data are sent to the database along with the image data, and the image is represented by a triangle with the shooting point as the apex. Image information management that enables image download by clicking a triangle displayed on the display by displaying the imaging range on a map and displaying it on the display, and also by linking the image data with the position data and storing it in the database. A system has been disclosed (eg, Patent Document 5).

WO2017/122660 WO2020/090229 WO2019/171777 Japanese Patent Application Laid-Open No. 2020-115082 JP 2004-118685 A

　Depending on the height of the structure to be imaged, it is necessary to change the height position and elevation angle (tilt) of the camera and shoot multiple fields of view with the same imaging point on the map. It is difficult to set and manage such three-dimensional information on a two-dimensional map. Also, if the location where a monitoring target such as a gas leak was detected is marked with an icon on the map, multiple monitoring targets with different height positions may be detected at the same position on the map, or there may be two types of monitoring targets. If there are more, it cannot be recognized from the map.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a monitoring support system capable of displaying icons attached to a large number of imaging fields in a monitoring area in an easy-to-understand three-dimensional manner. do.

The above problems of the present invention are solved by the following configuration.

(1) A monitoring support system in which an imaging device captures an image of a monitored object installed in a monitoring area and detects the monitored object from the obtained image,
a monitoring target coordinate estimating unit for estimating the spatial coordinates of a monitoring target included in the image, based on the spatial coordinates, azimuth angle, and elevation angle of the imaging device and the angle of view of the imaging device, which are input at the time of imaging;
a monitoring target distribution map generation unit that attaches an icon representing the monitoring target to the perspective view of the monitoring area based on the spatial coordinates of the monitoring target and outputs the result to a display device. support system.

(2) The monitoring support system according to (1), wherein the monitored object is stereoscopically displayed in the perspective view of the monitored area.

(3) A monitoring support system in which an imaging device captures an image of a monitored object installed in a monitoring area and detects the monitored object from the acquired image,
a monitoring target coordinate estimating unit for estimating the spatial coordinates of a monitoring target included in the image, based on the spatial coordinates, azimuth angle, and elevation angle of the imaging device and the angle of view of the imaging device, which are input at the time of imaging;
The object to be monitored is stereoscopically displayed on a plan view of the monitoring area, and an icon representing the object to be monitored is attached to a location on the object to be monitored on the plan view based on the spatial coordinates of the object to be monitored. , and a monitoring target distribution map generator for outputting to a display device.

(4) A monitoring support system in which an imaging device captures an image of a monitored object installed in a monitoring area and detects the monitored object from the acquired image,
a monitoring target coordinate estimating unit for estimating the spatial coordinates of a monitoring target included in the image, based on the spatial coordinates, azimuth angle, and elevation angle of the imaging device and the angle of view of the imaging device, which are input at the time of imaging;
A monitoring target distribution map in which an icon representing the monitoring target is added to each of the plan view of the monitoring area and the side view along the azimuth angle at locations based on the spatial coordinates of the monitoring target, and the monitoring target distribution map is output to a display device. A monitoring support system comprising: a generator;

(5) The monitoring support system according to (4), wherein the side view of the monitored area includes the side view of the monitored object.

(6) Based on the spatial coordinates, the azimuth angle, the elevation angle, and the angle of view of the imaging device, a figure indicating an imaging range in plan view and an imaging range in side view along the azimuth angle are displayed. Equipped with an imaging range figure generation unit that generates a figure,
The monitoring target distribution map generating unit attaches a figure indicating the imaging range in plan view to the plan view of the monitoring area, and attaches a figure indicating the imaging range in side view to the side view of the monitoring area. The monitoring support system according to (4) or (5), characterized in that the information is output to the display device as described above.

(7) an imaging range figure generation unit that generates a three-dimensional figure indicating an imaging range based on the spatial coordinates, the azimuth angle, the elevation angle, and the angle of view of the imaging device;
The monitoring target distribution map generation unit according to any one of (1) to (3), wherein the figure is attached to a plan view or a perspective view of the monitoring area and output to a display device. Surveillance support system.

(8) The monitoring support system according to any one of (1) to (7), wherein the acquired image is stored in association with an icon representing the monitoring target attached to the plan view or perspective view of the monitoring area. .

(9) Any of (1) to (8), including a display image generation unit that attaches an icon representing the monitored object to the acquired image in the margin of the image or on the monitored object, and outputs the image to a display device. A monitoring support system according to any one of the above.

(10) The display image generation unit attaches an icon representing a monitoring target included in another image obtained by imaging a subject outside the acquired image to a location based on the spatial coordinates of the monitoring target. and outputting to the display device (9).

(11) The monitoring support system according to (9) or (10), wherein the display image generation unit processes the acquired image, attaches the icon to the image, and outputs the image to the display device.

(12) The monitoring support system according to any one of (1) to (11), wherein the imaging device incorporates an inertial sensor and outputs an azimuth angle and an elevation angle during imaging.

(13) The monitoring support system according to any one of (1) to (12), wherein the imaging device incorporates positioning means and outputs spatial coordinates or plane coordinates during imaging.

According to the monitoring support system according to the present invention, it is possible to display icons attached to a large number of imaging fields in the monitoring area in an easy-to-understand three-dimensional manner.

1 is a block diagram illustrating the configuration of a monitoring support system according to an embodiment of the invention; FIG. It is a flowchart which shows the processing method by the monitoring assistance system which concerns on embodiment of this invention. 4A and 4B are a plan view and a side view of a monitoring area displayed on a display device of the monitoring support system for explaining processing by the monitoring support system according to the embodiment of the present invention; FIG. 4A and 4B are a plan view and a side view of a monitoring area displayed on a display device of the monitoring support system for explaining processing by the monitoring support system according to the embodiment of the present invention; FIG. 4A and 4B are a plan view and a side view of a monitoring area displayed on a display device of the monitoring support system for explaining processing by the monitoring support system according to the embodiment of the present invention; FIG. 4A and 4B are a plan view and a side view of a monitoring area displayed on a display device of the monitoring support system for explaining processing by the monitoring support system according to the embodiment of the present invention; FIG. 4A and 4B are a plan view and a side view of a monitoring area displayed on a display device of the monitoring support system for explaining processing by the monitoring support system according to the embodiment of the present invention; FIG. Composite image of visible light image with gas cloud and icon. FIG. 11 is an oblique projection view of a monitoring area displayed on the display device of the monitoring support system for explaining processing by the monitoring support system according to the modified example of the embodiment of the present invention; 5B is a partial enlarged view of the oblique projection view of the monitoring area shown in FIG. 5A; FIG.

A monitoring support system in a mode (embodiment) for carrying out the present invention will be described below with reference to the drawings. It should be noted that the size, positional relationship, etc. of the members shown in the drawings may be exaggerated for clarity of explanation, and the shape and structure may be simplified. Moreover, in the following description, the same reference numerals are given to the same or homogeneous members, and the description thereof will be omitted as appropriate.

[Monitoring support system]
A monitoring support system according to an embodiment of the present invention monitors leaked gas in a production facility or the like that produces natural gas. Monitor for leaks. As shown in FIG. 1 , the monitoring support system 10 includes a terminal device 1 having an imaging area diagram generation unit 4 and a display device 15 , an imaging device 2 having an infrared camera 21 , and a management device 7 . The terminal device 1 is communicably connected to the imaging device 2 by a cable or the like, or formed integrally, and provided in the monitoring support system 10 at least one (one set), and each user (worker) carries one. . The management device 7 communicates with each of one or more terminal devices 1 via a server or a cloud (not shown).

(imaging device)
The imaging device 2 includes an infrared camera 21 , a visible light camera 22 , an imaging control section 23 , an inertial sensor 24 , and a positioning section (positioning means) 25 . The infrared camera 21 is compatible with infrared rays in a wavelength band that is absorbed by hydrocarbon gas, and the visible light camera 22 is a general color or monochrome imaging device compatible with visible light. It is preferable that the infrared camera 21 and the visible light camera 22, especially the infrared camera 21, be capable of capturing moving images. The imaging device 2 is configured so that the infrared camera 21 and the visible light camera 22 can simultaneously image the same subject, and for example, respective lenses are arranged close to each other vertically on the front. The imaging control unit 23 is a CPU (Central Processing Unit) or a microcomputer, and controls the image sensors of the infrared camera 21 and the visible light camera 22, and the like. The inertial sensor 24 is an angular velocity sensor (gyro sensor) or an acceleration sensor, and observes the attitude (inclination) of the imaging device 2 . The positioning unit 25 is a GNSS (Global Navigation Satellite System) receiving module, and includes an antenna 25a. Get the location information (coordinates, latitude and longitude, altitude) of .

(Terminal device)
The terminal device 1 includes a control unit 11, a storage unit 12, a communication unit 13, an operation unit 14, a display device 15, a gas detection unit 31, a gas coordinate estimation unit (monitoring target coordinate estimation unit) 32, an imaging area display diagram generation unit ( A monitoring target distribution map generation unit 4 and a display image generation unit 5 are provided. A control unit 11 is a CPU. The storage unit 12 is a flash memory, a RAM (Random Access Memory), an HDD (Hard Disk Drive), or the like. The communication unit 13 is a unit composed of an antenna, an analog circuit, and the like for wireless communication such as Wi-Fi (registered trademark). The operation unit 14 is a keyboard or touch panel for inputting by a user (operator), and further includes a shutter button and the like when the terminal device 1 is formed integrally with the imaging device 2 . The display device 15 is a liquid crystal display or the like, and is preferably a touch panel integrated with the operation unit 14 . A mobile information terminal such as a tablet or a notebook PC can be applied to the terminal device 1 including these components. The gas detector 31, the gas coordinate estimator 32, the imaging area display diagram generator 4, and the display image generator 5 are each composed of a CPU, a RAM, etc., and are configured as part of the controller 11 and the storage unit 12. process.

The gas detection unit 31 detects an image of a substance suspected to be a gas from the difference between the infrared image and the visible light image captured by the imaging device 2 or the time difference between the infrared images, and furthermore, this image is the subject of the infrared image. It is determined whether the leaked hydrocarbon-based gas (hereinafter referred to as gas) from a structure such as a tank or other gas such as water vapor. Substances suspected of being gas are referred to herein as gaseous substances. The gas coordinate estimator 32 estimates the spatial coordinates of the gaseous substance detected by the gas detector 31 . These processes can be performed by the methods described in Patent Documents 1 to 4, for example.

The imaging area display diagram generation unit 4 generates a fan-shaped or triangular marker (imaging range ) and a map of the monitored area with icons indicating locations where gas leaks have occurred. The imaging area display diagram generation unit 4 includes a map generation unit 41 , an imaging range graphic generation unit 42 , an icon selection unit 43 , and an area diagram synthesis unit 44 . The map generator 41 generates a plan view (map) and a side view of the monitored area based on geographic information (spatial information) of the monitored area. Based on the coordinates of the current location and the orientation (pan, tilt) of the imaging device 2 received from the imaging device 2, the imaging range graphic generation unit 42 generates markers (imaging range graphics) and pointers indicating the position and orientation of the imaging device 2. It is generated with the coordinate information attached. The icon selection unit 43 selects an icon to be attached to a map or the like based on the detection result of the gas detection unit 31 or the user's operation. The area diagram synthesizing unit 44 synthesizes the marker and pointer generated by the imaging range figure generating unit 42 and the icon selected by the icon selecting unit 43 with the map and the side view generated by the map generating unit 41 .

The display image generation unit 5 combines the visible light image captured by the imaging device 2 with the gas cloud and icon in the infrared image to generate an image for display (hereinafter referred to as a composite image). The display image generator 5 includes a gas image processor 51 and an image synthesizer 52 . The gas image processing unit 51 extracts the gas cloud from the infrared image and performs image processing such as colorization so that it can be easily visually recognized. The image synthesis unit 52 synthesizes the gas cloud image processed by the gas image processing unit 51 and the icon selected by the icon selection unit 43 of the imaging area display diagram generation unit 4 into a visible light image. These processes can be performed by the methods described in Patent Documents 1 to 4, for example.

(Management device)
The management device 7 includes a control section 71, a database 72, and a communication section 73, and further includes an operation section 74 and a display device 75 as necessary. A control unit 71 is a CPU and controls a database 72 and a communication unit 73 . The database 72 is a HDD (Hard Disk Drive) or the like, and stores geographic information of the monitored area, infrared images and visible light images (collectively referred to as images as appropriate) received from the terminal device 1, and the like. The communication unit 73 is a unit that performs wireless communication, like the communication unit 13 of the terminal device 1 . The operation unit 74 is a keyboard or the like for a user (administrator) to make an input. The display device 75 is a liquid crystal display or the like. The management device 7 may be operated by an administrator, or may be operated by an operator carrying the terminal device 1 via the terminal device 1 .

The geographic information of the monitoring area includes at least a plan view of the monitoring area, specifically a map or a sketch, and further three-dimensional information ( three-dimensional shape). The geographic information of the monitored area is stored in the database 72 of the management device 7 or the storage unit 12 of the terminal device 1, or received from a GIS (Geographic Information System). Also, the map of the monitoring area may be acquired from GIS, and the three-dimensional information of the structure may be stored in the database 72 or the like. The three-dimensional information of the structure includes at least the height of the structure. In this case, for example, assuming that the shape of the structure is a pillar, a side view is generated by combining the height and the plan view (map). be able to.

〔Processing method〕
A processing method by the monitoring support system 10 according to this embodiment will be described with reference to FIGS. 2 and 3A to 3E. FIG. 2 is a flow chart showing a processing method mainly using the terminal device 1. As shown in FIG. 3A to 3E are examples of a plan view and a side view of the monitoring area displayed on the display device 15 of the terminal device 1. FIG.

In order to start monitoring the monitoring area, first, the user (worker) activates the terminal device 1 and inputs worker information such as a user ID (S11). Then, the communication unit 13 becomes ready for communication, and the geographic information of the monitoring area is received from the database 72 or GIS of the management device 7 (S12). Based on the geographic information, the map generator 41 generates a plan view (map) of the monitored area. In addition, when the geographic information is stored in the storage unit 12, reception from the outside is unnecessary. Further, the configuration may be such that the user ID or the like is input for starting the terminal device 1, and the input of the operator information (S11) is not executed.

In addition, the positioning unit 25 of the imaging device 2 sequentially acquires the position information of the current location, and the inertial sensor 24 observes the attitude of the imaging device 2, and these information are input to the terminal device 1 (S21). . Then, the imaging range graphic generation unit 42 generates arrow-shaped pointers p1 and p2 (see FIG. 3A). The pointer p1 has a position (planar coordinates) and an azimuth angle θ in a plan view, and the pointer p2 has a spatial coordinate including a height position and an elevation angle φ. The azimuth (pan) θ and elevation (tilt) φ of the imaging device 2 (infrared camera 21 and visible light camera 22) are appropriately collectively referred to as azimuth angles θ and φ. It should be noted that the angle of elevation (tilt) φ includes a range of less than 0°, that is, the angle of depression. Further, the map generation unit 41 generates a side view along the azimuth angle θ, which is an enlarged view of the vicinity of the current location based on the geographic information. Then, the area view synthesizing unit 44 attaches a pointer p1 to the map and a pointer p2 to the side view, and displays the map 81a and the side view 82a side by side on the display device 15 as shown in FIG. 3A (S22). . Here, pointers p1 and p2 indicate the current location at the tips of the arrows. The pointer p1 on the map 81a displayed on the display device 15 and the side view 82a with the pointer p2 are moved according to the movement of the imaging device 2 (S23, NO). In FIGS. 3A to 3E, the azimuth angle θ is assumed to be 0° (X-axis direction) for the sake of simplicity.

When the imaging device 2 is fixed on a tripod to capture an image of the structure 91 on the map 81a, and the imaging device 2 remains stationary for a predetermined time (for example, 2 seconds) (S23, YES), the imaging range figure generation unit 42 , fan-shaped markers (imaging range figures) m1 and m2 indicating the range that can be captured by the imaging device 2 at this time are represented by the spatial coordinates (imaging point) of the current location (imaging device 2), the azimuth angles θ and φ, and the imaging device 2 (infrared camera 21, visible light camera 22) and the angle of view (horizontal angle of view, vertical angle of view) and imaging distance (distance to a subject that can be focused). More specifically, the marker m1 indicates the imaging range in plan view, and is a fan-shaped circle whose radius is the imaging distance and whose central angle is the horizontal angle of view. The bisector is placed in the azimuth angle θ direction. Marker m2 indicates the imaging range in side view, and is a sector whose radius is the center angle of the circle with the imaging distance and the vertical angle of view. The angle bisector is placed in the elevation angle φ direction. Then, the area view synthesizing unit 44 puts a marker m1 on the map 81a (see FIG. 3A) displayed on the display device 15 and a marker m2 on the side view 82a, and displays them as shown in FIG. 3B (S24). ). Then, when the infrared camera 21 and the visible light camera 22 of the imaging device 2 capture the infrared image and the visible light image (S25), together with these images, the imaging date and time, the spatial coordinates and the azimuth angles θ and φ of the imaging device 2 are stored. (S26). Note that if the imaging device 2 resumes movement without imaging (S23, NO), the markers m1 and m2 are deleted from the map 81b and the side view 82b.

Based on the infrared image and the visible light image acquired by the imaging device 2, the gas detection unit 31 determines whether there is a substance suspected of being gas (gaseous substance) in the infrared image (S31). When a gaseous substance is detected (S31, YES), the gas coordinate estimator 32 estimates its spatial coordinates (S32). Determine (S33). If the gaseous substance is gas (S33, YES), the display image generator 5 combines the visible light image Vi with the gas cloud c1 as shown in FIG. 4 (S34). Further, the icon selection unit 43 selects the icon i1 if the gaseous substance is gas (S33, YES), and selects the icon i2 if the gaseous substance is water vapor (S33, NO). These icons i1 and i2 are displayed in the visible light image Vi by the display image generator 5, and the area diagram synthesizing unit 44 transforms them into the map 81b and the side view 82b (see FIG. 3B) displayed on the display device 15, respectively. Attached to the location corresponding to the estimated coordinates of the substance (S35, S36). In addition, the display image generation unit 5 stores the visible light image Vi obtained by synthesizing the gas cloud c1 and the icons i1 and i2 in the storage unit 12 as a synthesized image 84 (see FIG. 4). Although two types of icons i1 and i2 corresponding to gas and water vapor are displayed in FIGS. 3C and 4, for example, an icon may be displayed only when gas is detected. Alternatively, the operator may specify the type or presence/absence of the icon regardless of the determination by the gas detection unit 31 or in addition to the determination.

When the processes of S32 to S36 are completed, and when no gaseous substance is detected in the infrared image (S31, NO), as shown in FIG. , m2 are changed (S41). In this way, when monitoring in one field of view is completed, if there is a next imaging point (S42, YES), the imaging point is moved to execute imaging in a new field of view. When all imaging in the monitoring area is finished (S42, No), the monitoring work is completed.

It should be noted that the movement of the imaging device 2 includes cases where the plane coordinates are the same and only the azimuth angle θ or only the height or elevation angle φ is changed. For example, when only the elevation angle φ of the imaging device 2 is changed as shown in FIG. 3D, the position of the pointer p1 does not change in the map 81d compared to FIG. 3C, and the marker m3 is the marker m1 displayed in the same position as On the other hand, in the side view 81d, the orientation of the pointer p2 is changed, and a marker m4 is added by partially overlapping the marker m2 in the previous imaging. Note that, as in the map 81d, when the marker m1 indicating the imaging range for which imaging has been completed and the marker m3 to be imaged are present at the same plane coordinate, the marker It is preferred to display m3. Also, as shown in FIG. 3E, the icon i3 is hidden by the icons i1 and i2 on the map 81e and is not displayed, but is displayed on the side view 82e so that the difference in coordinates is easy to understand. If gaseous substances are detected in an area where the imaging fields of view overlap and their spatial coordinates are estimated to be the same, the gas detector 31 or the user determines whether they are the same gas or water vapor. , is the same, it is preferable to configure so that no new marker is added.

3A to 3E, maps 81a to 81e (referred to as map 81 or plan view 81 when not distinguished) and side views 82a to 82e (referred to as side view 82 when not distinguished) are shown. Although they are arranged horizontally and displayed, they may be arranged vertically according to the shape of the display device 15 or the like. Further, it may be configured such that only one of the map 81 and the side view 82 can be selected and displayed on the display device 15 by the user's operation. 3A to 3E, the map 81 displays the entire monitoring area. You can shift the displayed area, rotate the orientation, or return to full display. Alternatively, it may be configured such that the vicinity of the current location is automatically enlarged, or the azimuth angle of the imaging device 2 is rotated in a predetermined direction (for example, horizontal to the right) within the map 81 and displayed. good. Alternatively, if the imaging device 2 is fixed (S23, YES), the map 81 is displayed by enlarging and rotating the vicinity of the current location as described above, and if the imaging device 2 resumes movement (S23, NO, S42, YES), it may be configured to switch to the full area display.

In this way, by displaying not only a map (plan view) but also a side view, a large number of markers indicating the imaging field of view in the monitoring area and icons indicating the occurrence of abnormalities such as gas leaks are displayed in a spatially easy-to-understand manner. In addition, by changing the color of the marker after imaging (S41), the current location of the worker can be easily recognized even if many markers are attached on the map. Therefore, it is preferable to set the color of the marker so that the one at the current location stands out. Alternatively, the present location marker may be displayed blinking, or the current location marker may be displayed blinking before imaging. In addition, by displaying the field of view for which imaging has been completed on the map 81 and the side view 82, it is possible to prevent blank spaces that are not imaged and unnecessary duplication of imaging. Then, by selecting a marker on the map 81 or the side view 82, for example, by tapping the marker if the display device 15 is a touch panel, the composite image generated by the display image generation unit 5 can be displayed on the display device 15. Note that no gaseous substance is detected in the infrared image (S31, NO), and a visible light image is displayed in the field of view for which no composite image was generated. In addition, the displayed composite image can be displayed by switching the icon or the gas cloud to non-display, or can be displayed by switching the composite image to a visible light image or an infrared image by the user's operation. Alternatively, the icon of the map 81 or the side view 82 may be selected, and the composite image may be displayed on the display device 15 .

The markers and icons on the map 81 and the side view 82 may be configured so that display/non-display can be switched by user's operation. Also, the icons on the map 81 may be configured to be displayed when the scale of the map 81 is enlarged beyond a certain level. In addition, in the map 81 and the side view 82, along with displaying the icon, or instead of displaying the icon, the marker when the infrared image where the gaseous substance is detected is displayed in a different color from the marker where the gaseous substance is not detected. can also be displayed.

The shape of the pointer is not particularly defined as long as the coordinates and azimuth angle of the imaging device 2 are known. For example, a wedge shape or a simple picture of a camera may be applied. Alternatively, the marker may be displayed while the imaging device 2 is moving without using the pointer. Alternatively, the side view may be displayed after the imaging device 2 is fixed (S23, YES). The stationary determination (S23) of the imaging device 2 may be determined based on the position information of the current location acquired by the positioning unit 25, and may be configured to ignore changes in the posture of the imaging device 2. FIG. Further, for example, when the moving speed of the imaging device 2 is below a certain level or the moving distance per predetermined time is below a certain level, it may be determined to be stationary (S23, YES). Alternatively, the marker may be displayed at the same time as the pointer is displayed (S22) without executing the stillness determination (S23) of the imaging device 2. FIG.

The spatial coordinates and azimuth angles θ and φ of the imaging device 2, the image captured by the imaging device 2, the composite image synthesized by the display image generation unit 5, and the like are sequentially transmitted to the management device 7 and stored in the database 72. . Therefore, for example, when a plurality of workers are performing monitoring work at the same time, it is possible to display the field of view and the current location of other workers on the map 81 by using pointers and markers of different colors. Duplication can be prevented. In addition, by displaying the icon on the map 81, the gas leak detected by the terminal device 1 of another worker can be known. The display on the map 81 of the marker or the like indicating the imaging range by another worker can be configured to be switched by the operator's operation. The maps 81a to 81e shown in FIGS. 3A to 3E can also be displayed on the display device 75 of the management device 7 in real time. In addition, by summing up the entire imaging field of the monitoring area excluding overlap, the monitoring work time per imaging area and the number of detected gas leaks can be calculated, and the monitoring work cost can be calculated based on the imaging area. can be done.

When the imaging field of view in the monitoring area is set in advance, markers indicating the imaging range (planned imaging range) that has not yet been imaged (scheduled to be imaged) can be displayed on the map 81 in different colors. Also, if there is a scheduled imaging range in the displayed side view 82, it is preferable to display similar markers with different colors. Further, a marker is displayed on the map 81 to indicate the imaging range for which imaging has been completed. It is preferable to display the marker indicating the planned imaging range on the map 81 regardless of the positional relationship in the Z-axis direction. With such a configuration, omission of monitoring can be prevented. The planned imaging range is stored, for example, in the database 72 and received together with the geographic information of the monitored area (S12). Then, the map 81a (S22) displayed for the first time after the start of monitoring work is displayed with markers indicating all planned imaging ranges. Alternatively, a marker may be displayed on the map 81 that indicates the planned imaging range closest to the current location. Further, the display of the marker indicating the planned imaging range on the map 81 can be configured to be switched by the operator's operation. Further, when there are a plurality of scheduled imaging ranges with the same plane coordinates and azimuth angle θ but different heights and elevation angles φ, such markers are displayed on the map 81 in colors different from those of markers indicating other scheduled imaging ranges. may be displayed.

Maps

81a and 81b and

side views

82a and 82b are provided with icons and markers indicating gaseous substances or gas leaks detected in the previous monitoring work in the same imaging range as the imaging range at the current location. It is good also as a structure which displays. In other words, if the

maps

81a and 81b and the

side views

82a and 82b are not marked with icons, it means that no gas leak or the like was detected in the previous monitoring work. Alternatively, an icon or marker attached to the map 81 or the side view 82 may be selected to display an image captured in the previous or even previous monitoring work or a composite image.

The markers and icons attached to the map 81 and the side view 82 may be recorded in separate layers for each worker or each terminal device 1 (eg, machine No. 1, machine No. 2, etc.). Similarly, a marker indicating the imaging range for which imaging has been completed, a marker indicating the imaging range of the current location, and a marker indicating the planned imaging range may be recorded in separate layers. In this case, once the monitoring work is completed, it is recorded in one layer. With such a configuration, for example, an operator or an administrator can easily select a past date and display the map 81 or the like showing the monitoring results of that day on the display device 15 .

The icon in the composite image may be displayed in a predetermined area for annotation outside the visible light image. In addition, when a gaseous substance is detected in an infrared image in another field of view near the visual field of the visible light image, an icon is attached to the outer portion of the visible light image corresponding to the coordinates of this gaseous substance. may In addition, the display image generator 5 may synthesize an icon with the infrared image, and may emphasize or colorize the infrared image so that the gas cloud is easily visible. Further, the image to be combined with the icon is not limited to the unprocessed visible light image or infrared image (RAW data) captured by the imaging device 2. For example, it may be a colorized infrared image, or a high It may be a visible light image or an infrared image with finer and higher resolution. Therefore, the display image generation unit 5 has these image processing functions. Further, the imaging device 2 may be configured without the visible light camera 22 .

The positioning unit 25 of the imaging device 2 may be built into the terminal device 1. One or more of the gas detector 31 , the gas coordinate estimator 32 , the imaging area diagram generator 4 , and the display image generator 5 may be provided in the management device 7 instead of the terminal device 1 . In addition, the monitoring support system 10 may transmit and receive data such as current position information and captured images between a plurality of terminal devices 1 without including the management device 7 .

A user who carries the terminal device 1 and the imaging device 2 is not limited to a worker (human), and may be a robot, a drone, or the like. Such a terminal device 1 is program-controlled by the management device 7 or operated by an administrator via the management device 7 . In these cases, the terminal device 1 does not have to include the operation unit 14 and the display device 15 . Further, such a terminal device 1 and a terminal device 1 carried and operated by an operator (human) may be mixed.

The monitoring support system 10 according to the present embodiment is not limited to outdoor facilities such as production facilities that produce natural gas, but can also monitor the interior of a factory, for example. A sketch map, for example, can be used for such geographical information of the monitored area. The position information of the current location of the terminal device 1 can also be input by the operator at the time of imaging without being acquired by the positioning unit 25 of the imaging device 2 . For example, the operator taps the map 81 displayed on the display device 15 to input the plane coordinates, input the numerical value of the height position, or store it in advance according to the height of the tripod. Then, the azimuth angles θ and φ are acquired by the inertial sensor 24 . Also, the management device 7 and the terminal device 1, or between the terminal devices 1 may be connected by a wire such as a USB cable. Configuration. In addition, it is not limited to gas leaks, and liquid leaks in pipes and damages such as cracks in structures can also be monitored. The imaging device 2 corresponds to an object to be monitored. For example, the infrared camera 21 is not provided.

(Modification)
The monitoring support system 10 according to the above-described embodiment displays a plan view and a side view of the monitoring area, thereby displaying markers indicating the imaging field of view and icons indicating the occurrence of gas leaks and the like three-dimensionally in an easy-to-understand manner. A perspective view of the monitored area may be displayed. For example, as shown in FIGS. 5A and 5B, the monitored area is displayed in

oblique views

83c and 83c'. An oblique projection 83c is a three-dimensional view of the plan view (map) 81c shown in FIG. 3C, and an oblique projection 83c' is an enlarged view of the oblique projection 83c near the current position (pointer p1').

The

oblique projections

83c and 83c' show the plane view (map) of the monitoring area at an equal ratio without changing the direction of the XY plane, with the height direction (Z-axis direction) at 45°. Such oblique projection views 83c and 83c' are obtained by synthesizing the map of the monitoring area by the map generation unit 41 by representing the structure in a three-dimensional shape based on its three-dimensional information (height). Then, the imaging range graphic generation unit 42 generates the pointer p1' as an arrow oriented in accordance with the oblique projection views 83c and 83c', and generates the marker m1' as a three-dimensional shape. Then, the area view synthesizing unit 44 attaches icons i1 and i2 to the oblique projection views 83c and 83c' at positions corresponding to the estimated coordinates of the gaseous substance respectively indicated by the icons i1 and i2. Further, the oblique projection views 83c and 83c' may be expressed by emphasizing the length (height) in the Z-axis direction, for example.

In this way, by representing the monitoring area in a perspective view, it is possible to display markers indicating the imaging field of view and icons indicating the occurrence of gas leaks in a three-dimensional manner in an easy-to-understand manner. The side view 82c (see FIG. 3C) may be displayed on the display device 15 side by side with the oblique projection views 83c and 83c'. Further, a configuration may be adopted in which the oblique projection view 83c and the plan view 81c can be switched and displayed by the user's operation. The monitoring area can also be displayed in an overhead view such as an isometric view with the Z-axis direction as the vertical (vertical) direction. The pointer and marker are generated in a shape corresponding to the X-axis and Y-axis directions of the overhead view (perspective view).

The present invention is not limited to the above embodiments, and modifications can be made without departing from the scope of the present invention.

REFERENCE SIGNS LIST 10 monitoring support system 1 terminal device 11 control unit 12 storage unit 13 communication unit 14 operation unit 15 display device 2 imaging device 21 infrared camera 22 visible light camera 23 imaging control unit 24 inertial sensor 25 positioning unit (positioning means)
31 gas detector 32 gas coordinate estimator (monitoring target coordinate estimator)
4 Imaging area display map generation unit (monitoring target distribution map generation unit)
41 map generation unit 42 imaging range figure generation unit 43 icon selection unit 44 area map synthesis unit 5 display image generation unit 51 gas image processing unit 52 image synthesis unit 7 management device 71 control unit 72 database 73 communication unit 74 operation unit 75 display device

Claims

A monitoring support system in which an imaging device captures an image of a monitored object installed in a monitoring area and detects the monitored object from the obtained image,
a monitoring target coordinate estimating unit for estimating the spatial coordinates of a monitoring target included in the image, based on the spatial coordinates, azimuth angle, and elevation angle of the imaging device and the angle of view of the imaging device, which are input at the time of imaging;
a monitoring target distribution map generation unit that attaches an icon representing the monitoring target to the perspective view of the monitoring area based on the spatial coordinates of the monitoring target and outputs the result to a display device. support system.
The monitoring support system according to claim 1, wherein the monitored object is displayed three-dimensionally in the perspective view of the monitored area.
A monitoring support system in which an imaging device captures an image of a monitored object installed in a monitoring area and detects the monitored object from the obtained image,
a monitoring target coordinate estimating unit for estimating the spatial coordinates of a monitoring target included in the image, based on the spatial coordinates, azimuth angle, and elevation angle of the imaging device and the angle of view of the imaging device, which are input at the time of imaging;
The object to be monitored is stereoscopically displayed on a plan view of the monitoring area, and an icon representing the object to be monitored is attached to a location on the object to be monitored on the plan view based on the spatial coordinates of the object to be monitored. , and a monitoring target distribution map generator for outputting to a display device.
A monitoring support system in which an imaging device captures an image of a monitored object installed in a monitoring area and detects the monitored object from the obtained image,
a monitoring target coordinate estimating unit for estimating the spatial coordinates of a monitoring target included in the image, based on the spatial coordinates, azimuth angle, and elevation angle of the imaging device and the angle of view of the imaging device, which are input at the time of imaging;
A monitoring target distribution map in which an icon representing the monitoring target is added to each of the plan view of the monitoring area and the side view along the azimuth angle at locations based on the spatial coordinates of the monitoring target, and the monitoring target distribution map is output to a display device. A monitoring support system comprising: a generator;
The monitoring support system according to claim 4, wherein the side view of the monitored area includes the side view of the monitored object.
Based on the spatial coordinates, the azimuth angle, the elevation angle, and the angle of view of the imaging device, a figure indicating an imaging range in plan view and a figure indicating an imaging range in side view along the azimuth angle are generated. Equipped with an imaging range figure generation unit that
The monitoring target distribution map generating unit attaches a figure indicating the imaging range in plan view to the plan view of the monitoring area, and attaches a figure indicating the imaging range in side view to the side view of the monitoring area. 6. The monitoring support system according to claim 4, wherein the information is output to the display device.
an imaging range figure generation unit that generates a three-dimensional figure indicating an imaging range based on the spatial coordinates, the azimuth angle, the elevation angle, and the angle of view of the imaging device;
4. The monitoring target distribution map generation unit attaches the graphic to a plan view or a perspective view of the monitoring area and outputs the diagram to a display device. Surveillance support system.
The monitoring support system according to any one of claims 1 to 7, wherein the acquired image is stored in association with an icon representing the monitoring target attached to the plan view or perspective view of the monitoring area.
9. A display image generation unit for outputting an obtained image to a display device by attaching an icon representing the monitoring target outside the image or on the monitoring target to the acquired image. The monitoring support system according to .
The display image generation unit attaches an icon representing a monitoring target included in another image obtained by imaging a subject outside the acquired image to a location based on the spatial coordinates of the monitoring target, outside the obtained image, 10. The monitoring support system according to claim 9, which outputs to the display device.
11. The monitoring support system according to claim 9 or 10, wherein the display image generation unit performs image processing on the acquired image, attaches the icon to the image, and outputs the image to the display device.
The monitoring support system according to any one of claims 1 to 11, wherein the imaging device incorporates an inertial sensor and outputs an azimuth angle and an elevation angle during imaging.
The monitoring support system according to any one of claims 1 to 12, wherein the imaging device incorporates positioning means and outputs spatial coordinates or plane coordinates during imaging.