JP2024009427A - Inspection device, inspection method, and inspection program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an on-site inspection technique involving camera photography, which can identify a photographing position without introducing a special mechanism on-site.

SOLUTION: An inspection device 10A includes: an acquisition unit 21 that acquires a photographed image 11 of a site 30 while moving a camera 20; an extraction unit 22 that extracts a two-dimensional point group image 12 composed of feature points 15 from the photographed image 11; a creation unit 25 that creates a three-dimensional point group image 13 from a plurality of two-dimensional point group images 12 at different photographing positions P at the site 30; an estimation unit 26 that estimates, on the basis of the three-dimensional point group image 13, each photographing position P of the plurality of two-dimensional point group images 12; a derivation unit 27 that derives a parameter 18 that matches the three-dimensional point group image 12 to a reference map 17 of the site 30; and a recording unit 28 records, on the basis of the parameter 18, the photographing position P in association with position coordinates Q of the reference map 17.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2024,JPO&INPIT

Description

Embodiments of the present invention relate to on-site inspection techniques that involve camera photography.

At power plants and factories, inspectors regularly tour and perform maintenance and inspections on a vast amount of equipment to ensure safe operation. During their patrols, inspectors also visually check for any equipment abnormalities.

Traditionally, maintenance inspections were often done manually, with inspectors manually recording the results of their checks on site and creating inspection records at the office. To improve this situation, efforts are being made to utilize services that use mobile terminals to save labor. In such a service, an inspection procedure sheet is displayed on a screen for an inspector, and the inspector inputs the numerical values visually checked by the meter, etc. on the screen.

Furthermore, in such services, inspectors on patrol are equipped with cameras to record images like a drive recorder. At a later date, the images are checked as necessary to help discover or analyze equipment abnormalities.

Japanese Patent Application Publication No. 2008-146489 JP 2016-152003 Publication Japanese Patent Application Publication No. 2021-018710 JP2021-149420A

By the way, since the recorded videos taken during patrols last for a long time, there is a problem in that it takes time and effort to find the video of equipment suspected of being abnormal. Therefore, consideration is being given to linking the photographing position and the recorded video using positioning technology such as GPS, and easily confirming the video based on the location information of the device suspected of being abnormal. However, since radio waves cannot reach indoor areas such as power generation plants, it is difficult to specify the shooting position using GPS. It is also possible to introduce a special mechanism to specify the shooting position at the site where inspections are carried out, but the maintenance and upkeep costs of such a mechanism are an issue.

The embodiments of the present invention have been made in consideration of the above circumstances, and the purpose is to provide an on-site inspection technique that involves camera photography, which can identify the shooting position without introducing a special mechanism to the site. do.

In the inspection device according to the embodiment, an acquisition unit that acquires a captured image of the site while moving a camera, an extraction unit that extracts a two-dimensional point group image composed of feature points from the captured image, and a a creation unit that creates a three-dimensional point group image from the plurality of two-dimensional point group images having different photographing positions; and an estimation unit that estimates the photographing position of each of the plurality of two-dimensional point group images based on the three-dimensional point group image. , a deriving unit that derives parameters for matching the three-dimensional point group image with the reference map of the site, and a recording unit that records the photographing position in association with position coordinates of the reference map based on the parameters.

Embodiments of the present invention provide an on-site inspection technique that involves camera photography, which can specify a shooting position without introducing a special mechanism to the site.

FIG. 1 is a block diagram showing an inspection device according to a first embodiment of the present invention. (A) Image diagram of feature points that make up a two-dimensional point group image extracted by capturing images at different shooting positions while moving the camera on the scene, (B) Feature points extracted from multiple images taken at different shooting positions An image of the 3D point cloud image aligned with the reference map of the site. FIG. 2 is a block diagram showing an inspection device according to a second embodiment. 5 is a flowchart illustrating the steps of the inspection method and the algorithm of the inspection program according to the embodiment.

(First embodiment)
Embodiments of the present invention will be described below based on the accompanying drawings. FIG. 1 is a block diagram showing an inspection device 10A (10) according to a first embodiment of the present invention. FIG. 2(A) is an image diagram of feature points 15 constituting a two-dimensional point group image 12 extracted by capturing images at different shooting positions P (P ₁ , P ₂ ) while moving the camera 20 at the site 30. be. FIG. 2(B) is an image diagram in which a three-dimensional point group image 13 of feature points 15 extracted from a plurality of photographed images 11 at different photographing positions P (P ₁ to P ₅ ) is aligned with a reference map 17 of the site 30. .

As shown in FIG. 1, the inspection device 10A (10) includes an acquisition unit 21 that acquires a photographed image 11 of a site 30 while moving a camera 20, and a two-dimensional image composed of feature points 15 from this photographed image 11. an extraction unit 22 that extracts a point group image 12; a creation unit 25 that creates a three-dimensional point group image 13 from a plurality of two-dimensional point group images 12 having different photographing positions P on the site 30; An estimating unit 26 that estimates the photographing position P of each of the two-dimensional point group images 12; a deriving unit 27 that derives the parameters 18 for matching the three-dimensional point group images 12 with the reference map 17 of the site 30; and the photographing position based on the parameters 18. A recording unit 28 that records P in association with the position coordinates Q of the reference map 17 is provided.

The site 30 is assumed to be a power generation plant or factory that is regularly inspected by inspectors. Note that such patrols are not limited to being carried out by personnel, and may also be carried out by inspection robots. The inspector or the inspection robot carries a camera 20 that takes pictures of the surrounding area and the equipment to be inspected while moving.

Although the camera 20 shown in the embodiment is an omnidirectional camera, it is not limited to this, and a unidirectional camera can also be used. Furthermore, in addition to the case where the omnidirectional camera takes pictures while mechanically rotating the lens as shown in the figure, there are also cases where images are taken simultaneously with multiple ultra-wide-angle lenses and stitched together through image processing to form a single photographed image 11. . Furthermore, the photographed image 11 may also be a still image as well as a moving image.

The acquisition unit 21 may acquire the photographed images 11 of the site 30 in real time from the camera 20 that is moving on the site 30, or may acquire the captured images 11 of the site 30 all at once from the cameras 20 that have finished moving on the site 30. Then, the acquired photographed image 11 is superimposed on a celestial coordinate system defined for each photographing position P (fixed points P ₁ , P ₂ . . . on the flow line 36). Furthermore, the distortion of the photographed image 11 superimposed on the celestial coordinate system is corrected as necessary.

The extraction unit 22 extracts feature points 15 that have a remarkable contrast with the surroundings in the photographed image 11, and expresses the two-dimensional point group image 12 in each of the celestial coordinate systems defined for each fixed point (P ₁ , P _{2 .} . . ). do. Note that among the celestial coordinate systems of different photographing positions P, the same feature points 15 forming the two-dimensional point group image 12 are associated with each other.

The creation unit 25 creates a three-dimensional point group image 13 in the spatial coordinate system of the site 30 from the two-dimensional point group image 12 in _{the celestial coordinate system corresponding to each of the photographing positions P (fixed points P 1 ,} P 2 . . . on the flow line 36). Create. In addition. To create this three-dimensional point group image 13, information on the movement displacement (movement direction and movement distance) of the camera 20 is required. The displacement of this camera 20 can be measured by the provided acceleration sensor. Note that once the three-dimensional point group image 13 of the site 30 is created, it can be used repeatedly from the next time the site 30 is photographed, unless there is a change in the site 30 accompanied by a change in the photographed image 11.

The estimation unit 26 estimates the photographing position P of the two-dimensional point group image 12 from the relationship between the feature points 15 located in the three-dimensional point group image 13 and the feature points 15 located in the two-dimensional point group image 12. Thereby, in the spatial coordinate system of the site 30, the position of the photographing position P (fixed points P ₁ , P ₂ . . . on the flow line 36) is estimated.

The reference map 17 may be a design drawing (CAD drawing) of the site 30 or a three-dimensional image of the site 30 directly captured by laser. In this reference map 17, a portion that can become the flow line 36 of the photographing position P may be emphasized. Note that the reference map 17 is prepared in advance before the camera 20 starts photographing the site 30.

The derivation unit 27 derives parameters 18 that match the three-dimensional point group image 12 with the reference map 17 of the site 30. If the flow line 36 at the photographing position P has been emphasized in the reference map 17, highly accurate parameters 18 can be derived by matching this emphasized portion with the three-dimensional point group image 12.

The derived parameters 18 have the function of converting the coordinate system of the three-dimensional point group image 13 to the coordinate system of the reference map 17 or vice versa. As described above, when the three-dimensional point group image 13 of the site 30 is used repeatedly, the corresponding reference map 17 can also be used repeatedly, so the parameters 18 can also be used repeatedly.

The recording unit 28 converts the photographing position P defined in the coordinate system of the three-dimensional point group image 13 into the coordinate system of the reference map 17 based on the parameters 18 . Then, in this way, the position coordinates Q of the photographing position P defined in the coordinate system of the reference map 17 are recorded. Thereby, the corresponding photographed image 11 can be linked to the position coordinate Q of the reference map 17.

(Second embodiment)
Next, a second embodiment of the present invention will be described with reference to FIG. FIG. 3 is a block diagram showing an inspection device 10B (10) according to the second embodiment. The inspection device 10B of the second embodiment has a configuration in which a reading unit 31, a display unit 32, and a designation unit 35 are further added to the configuration of the inspection device 10A of the first embodiment described above. Note that in FIG. 3, parts having the same configuration or function as those in FIG.

Each photographed image 11 is associated with a respective photographing time 19 by the function of the camera 20. Then, the reading unit 31 reads the photographing time 19 from the acquired photographed image 11. As described in the first embodiment, the photographed image 11 is already linked to the photographing position P of the three-dimensional point group image 13 and the position coordinates Q of the reference map 17. Therefore, the recording unit 28 records the photographing time 19 in association with at least one of the photographing position P and the position coordinates Q.

The display unit 32 can display the reference map 17 and the photographed image 11 at the same time. The designation unit 35 can designate the position coordinates Q of the reference map 17 through operator operation. Thereby, the display unit 32 displays the photographed image 11 at the photographing position P corresponding to the designated position coordinate Q together with the reference map 17.

Further, the specifying unit 35 can specify the photographing time 19, and the display unit 32 can display the reference map 17 in which the position coordinates Q corresponding to the photographing time 19 are highlighted together with the photographed image 11. At this time, it is ideal that the photographing position P associated with an accurate actual measurement time and the position coordinate Q associated with the same photographing time 19 match. If the photographing position P and the position coordinates Q at the same time do not match, it is preferable to tune the functions of the estimation unit 26 and the derivation unit 27 so that they match. Thereby, the display accuracy of the position coordinates Q corresponding to the photographed image 11 displayed on the display section 32 can be improved in the reference map 17.

(Third embodiment)
The reference map 17 is classified into a three-dimensional image and a two-dimensional image, and since there is a difference in the function of the parameter 18 depending on the type, an example will be explained. When the reference map 17 is a three-dimensional image, the parameter 18 has a function of matching the reference map 17 and the three-dimensional point group image 13 through coordinate transformation that relatively rotates, translates, and changes the scale.

When the reference map 17 is a two-dimensional image, it has a function of converting the three-dimensional point group image 13 into a two-dimensional projected image. Furthermore, the parameter 18 has a function of aligning this projected image and the reference map 17 through coordinate transformation that relatively rotates, translates, and changes the scale. Even if the reference map 17 is a two-dimensional image, it can be treated as a three-dimensional image by adding height information of the moving route (flow line 36) of the camera 20 at the site 30.

The projected image of the three-dimensional point cloud image 13 onto the two-dimensional coordinate system is estimated in the horizontal direction from the recorded position information, and a two-dimensional drawing is generated from a point cloud at a certain height or a point cloud density in the vertical direction. Extract the point cloud to use. This makes it possible to exclude point cloud information that does not exist in the drawing, such as structures placed on the floor, and more robust parameters 18 are derived.

A more robust parameter 18 can also be derived by calculating the density of the number of points projected for each fixed area of the three-dimensional point group image 13 and projecting only the point group in areas with a specified density or higher onto the two-dimensional coordinate system. Ru. Further, a more robust parameter 18 can also be derived by calculating the average value of the three-dimensional point group image 13 in the direction perpendicular to the projection plane and extracting and projecting only the point group with values within a specified range from the average value.

Further, a marker (not shown) is placed at the scene 30, the photographing position P is estimated from the marker reflected in the photographed image 11, and the corresponding reference map 17 is automatically selected and transmitted to the deriving section 27. Good too.

The steps of the inspection method and the algorithm of the inspection program according to the embodiment will be explained based on the flowchart of FIG. 4 (see FIGS. 2 and 3 as appropriate). First, as shown in FIG. 2, the scene 30 is photographed while moving the camera 20 (S11), and a photographed image 11 is obtained (S12). Then, a two-dimensional point group image 12 composed of feature points 15 is extracted from each captured image 11 (S13).

Here, if the three-dimensional point group image 13 has not been created yet and is not present because it is the first patrol, etc. (S14, Yes), a three-dimensional point group image of the site 30 is created from these plural two-dimensional point group images 12. 13 is created (S15). Note that the creation of the three-dimensional point group image 13 additionally requires information on the movement displacement of the camera 20 measured by an acceleration sensor or the like provided in the camera 20. In this way, when the three-dimensional point group image 13 comes to exist (S14, No), the photographing position P of each of the plurality of two-dimensional point group images 12 is estimated based on this three-dimensional point group image 13. (S16).

Furthermore, if the parameter 18 has not yet been derived and is absent (S17, Yes), the parameter 18 that matches the three-dimensional point group image 12 with the reference map 17 of the site 30 is derived (S18). In this way, when the parameter 18 comes to exist (S17, No), the photographing position P is recorded in association with the position coordinates Q of the reference map 17 based on this parameter 18 (S19). Further, the photographing time 19 is read out from the photographed image 11 (S20) and linked to the photographing position P and position coordinates Q (S21).

Next, the reference map 17 is displayed (S22), and the position coordinates Q or the photographing time 19 are designated by an operator operation (S23). Then, the photographed image 11 at the corresponding photographing position P is displayed on the reference map 17 together with the position coordinates Q (S24). Then, the flow from (S22) to (S24) is repeated until the inspection of the site 30 is completed (S25, No, Yes, END).

According to the inspection device of at least one embodiment described above, by matching the reference map with a three-dimensional point group image of the site, which is made up of feature points extracted from camera images taken at a plurality of shooting positions, It becomes possible to specify each shooting position without introducing any special mechanism.

Although several embodiments of the invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, changes, and combinations can be made without departing from the gist of the invention. These embodiments and their modifications are included within the scope and gist of the invention as well as within the scope of the invention described in the claims and its equivalents.

The inspection device described above includes a control device with a highly integrated processor such as a dedicated chip, FPGA (Field Programmable Gate Array), GPU (Graphics Processing Unit), or CPU (Central Processing Unit), and a ROM (Read Only memory) and RAM (Random Access Memory), external storage devices such as HDD (Hard Disk Drive) and SSD (Solid State Drive), display devices such as displays, and input devices such as mice and keyboards. , and a communication I/F, and can be realized with a hardware configuration using a normal computer. Therefore, the components of the inspection device can be realized by a computer processor, and can be operated by an inspection program.

Further, the inspection program is provided by being pre-installed in a ROM or the like. Alternatively, this program is provided as an installable or executable file stored on a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD, or flexible disk (FD). You may also do so.

Further, the inspection program according to this embodiment may be stored on a computer connected to a network such as the Internet, and may be provided by being downloaded via the network. Furthermore, the inspection device can also be configured by combining separate modules that independently perform the functions of the constituent elements by interconnecting them via a network or a dedicated line.

Inspection device 10 (10A, 10B), camera 20, site 30, photographed image 11, two-dimensional point group image 12, three-dimensional point group image 13, feature points 15, reference map 17, parameters 18, photographing time 19, acquisition unit 21, Extraction unit 22, creation unit 25, estimation unit 26, derivation unit 27, recording unit 28, site 30, reading unit 31, display unit 32, designation unit 35, photographing position P, position coordinates Q.

Claims (10)

an acquisition unit that acquires a captured image of the scene while moving the camera;
an extraction unit that extracts a two-dimensional point group image composed of feature points from the photographed image;
a creation unit that creates a three-dimensional point cloud image from the plurality of two-dimensional point cloud images taken at different photographing positions at the site;
an estimation unit that estimates the photographing position of each of the plurality of two-dimensional point group images based on the three-dimensional point group images;
a derivation unit that derives parameters for matching the three-dimensional point cloud image with the reference map of the site;
An inspection device comprising: a recording unit that records the photographing position in association with position coordinates of the reference map based on the parameters. The inspection device according to claim 1,
a designation unit that designates the position coordinates of the reference map;
An inspection device comprising: a display unit that displays the photographed image at the photographing position corresponding to the specified position coordinates together with the reference map. The inspection device according to claim 2,
comprising a readout unit that reads out a photographing time from the photographed image,
The recording unit is an inspection device that records the photographing time in association with at least one of the photographing position and the position coordinates. The inspection device according to claim 3,
The designation unit can designate the photographing time,
The display unit is an inspection device that displays the reference map in which the position coordinates corresponding to the photographing time are highlighted together with the photographed image. In the inspection device according to claim 1 or claim 2,
The camera is an inspection device that is an omnidirectional camera. In the inspection device according to claim 1 or claim 2,
The inspection device aligns the parameters by coordinate transformation that relatively rotates, translates, and scales the reference map, which is a three-dimensional image, and the three-dimensional point group image. The inspection device according to claim 6,
The reference map is an inspection device that obtains the three-dimensional image by adding height information of the moving route of the camera to the information of the two-dimensional image. In the inspection device according to claim 1 or claim 2,
The inspection device aligns the reference map, which is a two-dimensional image, and the projection image, which is a two-dimensional projection image of the three-dimensional point group image, through coordinate transformation that relatively rotates, translates, and changes the scale of the reference map, which is a two-dimensional image. acquiring a captured image of the scene while moving the camera;
extracting a two-dimensional point group image composed of feature points from the photographed image;
creating a three-dimensional point group image from the plurality of two-dimensional point group images taken at different photographing positions at the site;
estimating the photographing position of each of the plurality of two-dimensional point group images based on the three-dimensional point group images;
deriving parameters for matching the three-dimensional point cloud image to the reference map of the scene;
An inspection method comprising the step of recording the photographing position in association with the position coordinates of the reference map based on the parameters. to the computer,
acquiring a captured image of the scene while moving the camera;
extracting a two-dimensional point group image composed of feature points from the photographed image;
creating a three-dimensional point group image from the plurality of two-dimensional point group images taken at different photographing positions at the site;
estimating the photographing position of each of the plurality of two-dimensional point group images based on the three-dimensional point group images;
deriving parameters for matching the three-dimensional point cloud image to the reference map of the scene;
An inspection program that executes a step of recording the photographing position in association with position coordinates of the reference map based on the parameters.

Images