WO2018155590A1

WO2018155590A1 - Identifying device, identifying method and program for identifying position of wall surface inside tunnel appearing in photographic image

Info

Publication number: WO2018155590A1
Application number: PCT/JP2018/006576
Authority: WO
Inventors: 緑川　克美; 和田　智之; 徳人斎藤; 加瀬　究; 隆士道川; 祐一小町; 幸太郎岡村; 武晴村上; 亨男坂下; 繁木暮
Original assignee: 国立研究開発法人理化学研究所
Priority date: 2017-02-24
Filing date: 2018-02-22
Publication date: 2018-08-30
Also published as: JPWO2018155590A1; JP7045721B2

Abstract

In an identifying device (101), a first acquiring unit (111) acquires a plurality of photographic images obtained by capturing images of a wall surface inside a tunnel. A first calculating unit (112) calculates a position of a feature point and a local feature quantity in the photographic images. A constructing unit (113) constructs a three-dimensional polygonal model of the wall surface to generate a wall surface map. A first mapping unit (114) associates each pixel in the photographic images with a position in the wall surface map. A second acquiring unit (121) acquires a search image obtained by newly capturing an image of the wall surface. A second calculating unit (122) calculates a position of a feature point and a local feature quantity in the search image. A second mapping unit (124) compares the positions of the feature points and the local feature quantities in the photographic images and the search image to associate each pixel in the search image with one of the pixels in the photographic images. An output unit (125) outputs the positions in the wall surface map to which the pixels in the photographic images associated with each pixel in the search image are associated.

Description

IDENTIFICATION DEVICE, IDENTIFICATION METHOD, AND PROGRAM FOR IDENTIFYING THE LOCATION OF A WALLEN IN A TUNNEL IN A PHOTOGRAPH IMAGE

The present invention relates to an identification device, an identification method, and a program for identifying the position of a wall surface in a tunnel reflected in a photographic image.

調査 Conventionally, investigation devices for grasping the condition of the wall surface of the tunnel have been proposed.

For example, in Patent Document 1,
Mount multiple video cameras facing the wall to the camera installation stand so that the distance to the wall is the same and the end of the field of view of the adjacent video camera overlaps on a plane perpendicular to the wall. By photographing the wall surface while moving the measurement vehicle while maintaining a substantially constant distance from the wall surface,
By facing the video camera directly to the wall surface, equalizing the distance to the wall surface and overlapping the ends of the imaging field of view, a wall image with a uniform scale is taken,
Use data measured by INS (Inertial Navigation System), odometer, and laser scanner when the image wrap rate is low or when the distance between the imaging means and the side surface changes and the scale of the image changes. Thus, a technique has been proposed that enables joining and creates a developed image with a uniform scale.

Non-Patent Document 1 discloses SfM (Structure from Motion) technology that acquires 3D information of a target from a plurality of 2D images or 2D moving images taken from different positions of the 3D target. ing.

By applying the technique disclosed in Non-Patent Document 2 to the wall surface image obtained by the technique disclosed in Patent Document 1, information on the three-dimensional shape of the wall surface in the tunnel can be obtained. Furthermore, by applying the technique disclosed in Non-Patent Document 3, a two-dimensional development view of the wall surface in the tunnel can be obtained.

Here, in order to find cracks, water leaks, cavities, etc. in the tunnel, it is necessary to conduct periodic inspections and repair them appropriately.

JP 2004-12152 A

However, since GPS (Global Positioning System) cannot be used in the tunnel, it is necessary to appropriately identify the position where the deformation is found on the wall of the tunnel. In particular, if the inspection and repair are performed on different days, whether or not the position of the deformity previously discovered by the inspector and the position currently observed by the repairer match. Therefore, it is required to improve work efficiency by determining at high speed whether there is a proper positional relationship.

In addition, the inspection method should be such that the entire tunnel wall surface is observed at a low frequency (for example, every 5 or 10 years), and follow-up observations of previously discovered deformations are performed at a high frequency (for example, every year). There are also many. In the follow-up observation of the deformation, it is necessary to identify the position that the inspector is currently observing, grasp the positional relationship with the deformation to be observed, and guide the inspector to the target location.

In addition, when a deformation is found on the wall of the tunnel, the position where the deformation is found is identified at high speed from the image of the deformation and the information of the tunnel that was previously inspected. There is a strong demand for technology to do this.

This invention solves said subject, and relates to the identification apparatus, the identification method, and program which identify the position of the wall surface in the tunnel reflected in the photograph image.

The identification device according to the present invention is:
Acquire multiple photographic images of the walls in the tunnel,
Calculating the position of each feature point and local feature amount of each of the plurality of photographic images;
A three-dimensional polygon model of the wall surface is constructed with reference to the calculated feature point positions and local feature amounts for each of the plurality of photographic images, and a wall map based on the constructed three-dimensional polygon model is generated. ,
Associating each pixel of the plurality of photographic images with a position in the wall map,
Obtain a search image in which the wall surface in the tunnel is newly photographed,
Calculating the position of the feature point of the acquired search image and the local feature amount;
By comparing the calculated feature point position and local feature amount for the search image with the calculated feature point position and local feature amount for each of the plurality of photographic images, Associating each pixel with one of the pixels of the plurality of photographic images,
A position in the wall surface map in which any pixel of the plurality of photographic images associated with each pixel of the search image is associated is output.

According to the present invention, it is possible to provide an identification device, an identification method, and a program for identifying the position of the wall surface in the tunnel reflected in the photographic image.

It is explanatory drawing which shows schematic structure of the identification apparatus which concerns on embodiment of this invention. It is a flowchart which shows the flow of control of an initialization process. It is a drawing-substituting photograph in which a plurality of photographic images taken on the wall of a tunnel are represented in 256 gray scales. It is a drawing-substituting photograph in which a plurality of photographic images taken on the wall of a tunnel are represented in two-tone monochrome. It is a drawing-substituting photograph showing a point group obtained by SfM in a gray scale of 256 gradations. It is a drawing-substituting photograph in which the point cloud obtained by SfM is expressed in two-tone monochrome. It is a drawing substitute photograph that represents a three-dimensional polygon model in 256 gray scales. It is a drawing substitute photograph that represents a three-dimensional polygon model in two-tone monochrome. It is a drawing substitute photograph that represents the vertices of a 3D polygon model in 256 gray scales. It is a drawing substitute photograph that represents the vertex of a 3D polygon model in two-tone monochrome. It is explanatory drawing explaining a mode that the point of a 3-dimensional polygon model is projected on a 2-dimensional expansion | deployment figure. It is the drawing substitute photograph which represents the development of the wall surface in the tunnel in 256 gray scales. It is a drawing-substituting photograph showing the development of the wall surface inside the tunnel in monochrome with two gradations. It is a flowchart which shows the flow of control of an identification process. In Experiment 1, it is a drawing substitute photograph that represents an example of a search image in a gray scale of 256 gradations. In Experiment 1, it is a drawing substitute photograph that represents an example of a search image in monochrome with two gradations. In Experiment 1, it is a drawing-substituting photograph that represents a photographic image that matches the search image with a gray scale of 256 gradations. In Experiment 1, it is a drawing-substituting photograph that represents a photographic image that matches the search image in two-tone monochrome. FIG. 5 is a drawing-substituting photograph that represents a three-dimensional polygon model in which a search image is searched in Experiment 1 with a gray scale of 256 gradations. FIG. 3 is a drawing-substituting photograph that represents a three-dimensional polygon model in which a search image is searched in Experiment 1 in two gradations of monochrome. In Experiment 2, it is a drawing substitute photograph that represents an example of a photographic image A of a crack to be searched in 256 gray scales. In Experiment 2, it is a drawing substitute photograph that represents an example of a photographic image B of a crack to be searched in 256 gray scales. In Experiment 2, it is a drawing substitute photo that represents an example of a photographic image C of a crack to be searched in 256 gray scales. In Experiment 2, it is a drawing substitute photograph that represents an example of a photographic image D of a crack to be searched in 256 gray scales. In Experiment 2, it is a drawing-substituting photograph in which an example of a cracked photographic image E to be searched is represented by a gray scale of 256 gradations. FIG. 5 is a drawing-substituting photograph that represents an example of a cracked photographic image A to be searched in Experiment 2 in two-tone monochrome. In Experiment 2, it is a drawing substitute photograph that represents an example of a photographic image B of a crack to be searched in monochrome of two gradations. In Experiment 2, it is a drawing-substituting photograph that represents an example of a cracked photographic image C to be searched in two-tone monochrome. In Experiment 2, it is a drawing substitute photograph that represents an example of a cracked photographic image D to be searched in two-tone monochrome. In Experiment 2, it is a drawing-substituting photograph that represents an example of a cracked photographic image E to be searched in two-tone monochrome. In Experiment 2, it is a drawing-substituting photograph in which the search image for the photograph image A is represented by monochrome of two gradations. In Experiment 2, it is a drawing-substituting photograph in which the search image for the photographic image B is represented in monochrome with two gradations. In Experiment 2, it is a drawing-substituting photograph in which the search image for the photograph image C is represented in two gradations of monochrome. In Experiment 2, it is a drawing-substituting photograph in which the search image for the photographic image D is represented in monochrome with two gradations. In Experiment 2, it is a drawing-substituting photograph in which the search image for the photographic image E is represented by two-tone monochrome. In Experiment 2, a photographic image A that matches the search image A is a drawing-substituting photograph that is expressed in two-tone monochrome. In Experiment 2, a photographic image B that matches the search image B is a drawing substitute photo that represents monochrome in two gradations. In Experiment 2, a photographic image C matching the search image C is a drawing-substituting photograph that represents two-tone monochrome. In Experiment 2, a photographic image D that matches the search image D is a drawing-substituting photograph that is expressed in two-tone monochrome. In Experiment 2, a photographic image E that matches the search image E is a drawing-substituting photograph that represents two-tone monochrome. In Experiment 2, a photographic image A that matches the search image A is a drawing substitute photo that represents a gray scale of 256 gradations. In Experiment 2, a photographic image B that matches the search image B is a drawing substitute photo that represents a gray scale of 256 gradations. In Experiment 2, a photographic image C matching the search image C is a drawing-substituting photo that represents a gray scale of 256 gradations. In Experiment 2, a photographic image D that matches the search image D is a drawing substitute photo that represents a gray scale of 256 gradations. In Experiment 2, a photographic image E that matches the search image E is a photo that substitutes for a drawing expressed in 256 gray scales. In Experiment 2, it is a drawing-substituting photograph that is obtained by enlarging the search result of the search image A-E in the development of the wall surface in the tunnel and expressing it in 256 gray scales. In Experiment 2, it is a drawing-substituting photograph that expands the search result of the search image A-E in the development view of the wall surface in the tunnel and expresses it in monochrome with two gradations. FIG. 5 is a drawing-substituting photograph that represents a three-dimensional polygon model in which a search image A-E is searched in Experiment 2 with 256 gray scales. FIG. 5 is a drawing-substituting photograph that represents a three-dimensional polygon model in which a search image A-E is searched in Experiment 2 in two-tone monochrome.

Hereinafter, embodiments of the present invention will be described. In addition, this embodiment is for description and does not limit the scope of the present invention. Accordingly, those skilled in the art can employ an embodiment in which each element or all elements of the present embodiment are replaced with equivalent ones. In addition, the elements described in each embodiment can be omitted as appropriate according to the application. As described above, any embodiment configured according to the principle of the present invention is included in the scope of the present invention.

(Overview configuration)
FIG. 1 is an explanatory diagram showing a schematic configuration of an identification apparatus according to an embodiment of the present invention. Hereinafter, an outline will be described with reference to this figure.

As shown in the figure, the identification apparatus 101 according to the present embodiment is realized by executing a predetermined program in a computer, and includes a first acquisition unit 111, a first calculation unit 112, a construction unit 113, and a first mapping unit. 114, a second acquisition unit 121, a second calculation unit 122, a second mapping unit 124, and an output unit 125.

Here, the first acquisition unit 111 acquires a plurality of photographic images taken of the wall surface in the tunnel.

A plurality of photographic images are taken by a video camera or a still camera as disclosed in Patent Document 1, for example.

On the other hand, the first calculation unit 112 calculates the position of each feature point and the local feature amount of a plurality of photographic images.

In an aspect to which the SfM technology disclosed in Non-Patent Document 1 is applied, local feature quantities such as SIFT (Scale-Invariant Feature Transform) and SURF (Speeded Up Up Robust Feature) are used.

Further, the construction unit 113 constructs a 3D polygon model of the wall with reference to the feature point positions and local feature amounts calculated for each of the plurality of photographic images, and a wall map based on the constructed 3D polygon model Is generated.

For example, the SfM technology disclosed in Non-Patent Document 1 is used to construct a three-dimensional polygon model.

In the SfM technology, from a plurality of photographic images, point cloud data representing the three-dimensional position of the feature points, photographic data representing which photographic image was photographed at which photographing position and in which photographing direction, Is estimated and output.

Here, since the distribution of the feature points included in the point cloud data is dense and dense, in order to specify the three-dimensional position with respect to an arbitrary pixel of the photographic image, for example, the curved surface reconstruction disclosed in Non-Patent Document 2 is performed. Do.

Here, as the wall surface map, a 3D map expressed by the 3D polygon model can be adopted, or a 2D map expressed by an expanded view of the 3D polygon model can be adopted. it can. For example, the technique disclosed in Non-Patent Document 3 can be applied to the generation of the two-dimensional map.

Then, the first mapping unit 114 associates each pixel of the plurality of photographic images with a position in the wall surface map. This association is called “first mapping”.

The processing performed by the first acquisition unit 111, the first calculation unit 112, the construction unit 113, and the first mapping unit 114 described above may be referred to as “initialization”. It is executed when the inspection is performed.

The position and local feature amount of each feature point in the plurality of photographic images calculated at the time of initialization and the wall surface map are used to search the search image and identify the position in the processing described later. It is stored in a recording medium, hard disk, database, etc.

After the initialization is completed, the process for performing the inspection after narrowing down the target portion with a relatively high frequency is as follows: the second acquisition unit 121, the second calculation unit 122, the second mapping This is executed by the unit 124 and the output unit 125. These processes may be collectively referred to as “search”.

That is, the second acquisition unit 121 acquires a search image in which a wall surface in the tunnel is newly photographed.

The plurality of photographic images acquired by the first acquisition unit 111 are photographed so as to cover the wall surface in the tunnel, but the search image acquired by the second acquisition unit 121 It is the photograph which image | photographed the position in the wall surface which is paying attention.

On the other hand, the second calculation unit 122 calculates the position of the feature point and the local feature amount of the acquired search image.

The second calculation unit 122 calculates the position of the feature point and the local feature amount using the same algorithm as the first calculation unit 112.

Further, the second mapping unit 124 compares the position of the feature point and the local feature amount calculated for the search image with the position of the feature point and the local feature amount calculated for each of the plurality of photographic images. Each pixel of the search image is associated with any pixel of the plurality of photographic images.

Specifically, the second mapping unit 124 selects feature points in a plurality of photographic images having local feature amounts similar to the local feature amount for each feature point in the search image, and each feature point in the search image Is associated with one of feature points in a plurality of photographic images. Positions other than feature points are associated by using Delaunay triangulation. This association is referred to as “second mapping”.

Then, the output unit 125 outputs a position in the wall map in which any pixel of the plurality of photographic images associated with each pixel of the search image is associated.

That is, the output unit 125 obtains positions in a plurality of photographic images taken at the time of initialization by applying the second mapping to the positions of the respective pixels of the search image. Then, by applying the first mapping to the position, the position on the wall surface map of each pixel of the search image is obtained.

The initialization process and the identification process can be executed at different times and frequencies. Therefore, the computer that executes the initialization process may be the same as or different from the computer that executes the identification process. Therefore, the program for executing the initialization process and the program for executing the identification process may be prepared together as one program, or may be prepared as independent programs.

Typically, the program is recorded on a recording medium, loaded into a memory included in the computer, and executed by a processor included in the computer. The program is expressed as a collection of codes for realizing each unit of the identification apparatus 101.

In addition, it is possible to speed up the calculation by efficiently using GPU (Graphics-Processing-Unit). For example, the coordinates for each pixel of a plurality of photographic images can be easily calculated by using the OpenGL rendering function.

That is, the first mapping unit 114 converts the three-dimensional coordinates themselves of each point of the three-dimensional polygon model or the two-dimensional coordinates in the development view of each point into color information, and the obtained color information is converted to each point. Build a three-dimensional colored model by assigning to the colors.

Then, the first mapping unit 114 renders a three-dimensional color model from the shooting position and shooting direction estimated by SfM. If the rendering result is the same size as each photographic image, the color information of the pixel arranged at the same position as each pixel in each photographic image is obtained in the rendering result, and if the inverse conversion to coordinates is performed, Coordinate information can be obtained.

The color information of each pixel can be calculated with floating-point precision by using OpenGL FBO (Frame Buffer Object), etc., and can be calculated using GPU, so high precision coordinate values are fast and robust It becomes possible to calculate.

In addition, if a technology such as FPGA (Field Programmable Gate Array) is applied, the program can be used as a design drawing of an electronic circuit. In this aspect, an electronic circuit for performing initialization processing or an electronic circuit for performing identification processing is realized as hardware based on the program.

In the following, for ease of understanding, an example will be described in which the identification apparatus 101 is realized as a whole by executing an initialization program and an identification program by a computer.

(Initialization)
FIG. 2 is a flowchart showing the control flow of the initialization process. Hereinafter, a description will be given with reference to FIG.

When the copy process is started, the computer that executes the initialization program first acquires a plurality of photographic images obtained by photographing the wall surface in the tunnel (step S201).

Fig. 3 is a drawing substitute photo showing a plurality of photographic images taken on the wall of the tunnel in 256 gray scales. FIG. 4 is a drawing-substituting photograph in which a plurality of photographic images taken on the wall surface of the tunnel are represented in two-tone monochrome. The illustrated image is a part of a plurality of photographic images acquired in step S201. As these photographic images, each frame of a moving image taken while rotating a vehicle-mounted video camera may be used, or an image taken while moving and rotating a still camera may be used.

Next, the computer estimates the shooting position and shooting direction of the camera in the three-dimensional point group and each photographic image from the plurality of photographic images using the feature point group included in these by SfM (step S202). . FIG. 5 is a drawing-substituting photograph showing the point group obtained by SfM in a gray scale of 256 gradations. FIG. 6 is a drawing-substituting photograph in which the point group obtained by SfM is expressed in two-tone monochrome.

In SfM, as an estimation result, for each point included in the three-dimensional point group, the three-dimensional coordinate value of each point, the shooting position and shooting direction when each point is shot in each photographic image, and Are output in association with each other.

Next, the computer uses the curved surface reconstruction method to generate polygons passing through the three-dimensional point group, construct a three-dimensional polygon model, and generate a wall surface map based on the three-dimensional polygon model (step S203). Since the point cloud is a three-dimensional representation of feature points in a photographic image, the three-dimensional coordinates of any point on the wall surface in the tunnel can be estimated by reconstructing the curved surface.

In saddle surface reconstruction, a method is widely used in which point cloud data with a normal is input, a scalar field with a sign is defined by an implicit function surface, and its isosurface is extracted. FIG. 7 is a drawing-substituting photograph that represents the three-dimensional polygon model in 256 gray scales. FIG. 8 is a drawing-substituting photograph that represents a three-dimensional polygon model in monochrome with two gradations.

In this figure, a three-dimensional polygon model generated for a wall surface in a tunnel photographed in a photographic image by the Smooth-signed Distance-Surface-Reconstruction method disclosed in Non-Patent Document 2 is shown.

Since each polygon is constructed so as to pass through the three-dimensional point group, the three-dimensional point group and the polygon are associated with each other by identity transformation mapping.

The generated 3D polygon model is a 3D map that expresses the 3D coordinate values of each point on the wall in the tunnel as it is. This three-dimensional map can be used as a wall map.

Furthermore, if necessary, a developed view of the wall surface in the tunnel can be used as a wall surface map. The two-dimensional coordinate value of each point on the wall surface in the tunnel is expressed in the development view.

Development generation results in a parameterization problem that embeds a 3D polygon in 2D. For example, in the Mean-Value-Coordinates method disclosed in Non-Patent Document 3, a two-dimensional coordinate value is assigned to each vertex of a three-dimensional polygon. In the Mean-Value-Coordinates method, polygon data vertices are classified into boundary vertices and internal vertices for parameterization.

First, of the boundary vertices, the corner vertices are arranged at the positions of the polygon corners. Other boundary vertices are arranged on a straight line using the length of the boundary ridgeline. FIG. 9 is a drawing-substituting photograph showing vertices of a three-dimensional polygon model in a gray scale of 256 gradations. FIG. 10 is a drawing-substituting photograph in which the vertex of the three-dimensional polygon model is represented in two-tone monochrome. On the left side of the figure, the cross section of the polygon P is drawn in an arc shape, and the end point corresponds to the boundary vertex.

On the other hand, the internal vertex v _i is subjected to local parameterization by linear combination using its neighboring vertices v _{i, j} .

座標 The coordinate values after parameterization are simultaneous equations with the internal vertex coordinate values as unconstant. By solving this, the mapping from the polygon to the development can be defined.

With this mapping, two-dimensional coordinate s _j and 3-dimensional coordinates p _j of the input image M _i on the pixels (pixel) m _j may calculated as follows. FIG. 11 is an explanatory diagram for explaining how the points of the three-dimensional polygon model are projected onto a two-dimensional development view.

First, the imaging position C _i of the image M _i, and pixel m _j, the line of sight is defined. The position of the intersection of this line of sight and the polygon P is the three-dimensional coordinate p _j . Hereinafter, in order to facilitate understanding, the point itself is appropriately represented by coordinates.

Then, to get the triangle t _k including intersection p _j, by taking a weighted average of the vertices of the triangle t _k, obtaining the barycentric coordinates of the triangle t _k.

Finally, by combining the obtained barycentric coordinates and the vertex coordinate values of the triangle t _k projected on the development view, the coordinate values s _j on the development view for the intersection point p _j can be acquired. FIG. 12 is a drawing-substituting photograph showing a development view of the wall surface in the tunnel in a gray scale of 256 gradations. FIG. 13 is a drawing-substituting photograph in which the development of the wall surface in the tunnel is shown in two-tone monochrome.

Thus, the computer image M _i in the pixel m _j 3-dimensional coordinates p _j in the three-dimensional polygon model for a 2-dimensional coordinate s _j in or a two-dimensional development view, is calculated to the first mapping Obtain (step S204).

Then, the computer stores the association based on the calculation result as a first mapping that associates the wall surface map with each pixel in a memory or a hard disk, and also stores the positions of the feature points and local feature amounts of a plurality of photographic images in the memory or hard disk (Step S205), and the process is terminated.

(Identification process)
As described above, the initialization process and the identification process are typically executed at different times. In the identification process, it is identified which position in the wall map to be constructed by the initialization process the search image obtained by photographing the wall surface in the tunnel corresponds to. FIG. 14 is a flowchart showing a flow of control of the identification process. Hereinafter, a description will be given with reference to FIG.

Identification processing is realized by a computer executing a program for identification processing. When the identification process is started, first, the computer acquires a search image (step S301).

From the point of view of the inspector, it is desirable that when the search image is taken, the position of the taken region in the wall map is immediately determined. For this purpose, the search image is desirably acquired from a camera directly connected to a computer.

In other words, each time the searcher takes a search image with the camera, the inspector can obtain which point in the wall map is currently being observed. Also, when a crack or the like is found, if it is photographed, it is possible to identify whether this is a newly generated deformation or a deformation that has been discovered in the past.

Next, the computer calculates the position of the feature point and the local feature amount in the acquired search image (step S302). The same algorithm is used in the initialization process and the identification process for calculating the position of the feature point and the local feature amount. As mentioned above, SIFT, SURF, etc. can be adopted.

Then, the computer compares the feature point positions and local feature amounts calculated for the search image with the feature point positions and local feature amounts calculated for a plurality of photographic images in the initialization process. Then, a photographic image that matches the search image is searched from a plurality of photographic images (step S303). With this process, a second mapping representing the correspondence between the pixels of the search image and the matching image is obtained.

The bag search is performed as follows. First, a feature point pair obtained by combining each feature point calculated for a search image and any one of feature points calculated for a plurality of photographic images is searched.

That is, one feature point pair is a pair of one feature point in the search image and one feature point in any one of a plurality of photographic images, and these two feature points. The local feature values calculated for are similar to each other.

Then, a photographic image in which many feature points obtained by the feature point pair appear is regarded as an image that matches the search image.

特徴 Although the feature point pair may include an illegal pair, the illegal pair can be removed by using the RANSAC method.

In the coordinate conversion using the RANSAC method, first, four feature points are selected at random from one image, the feature points of the other image paired with these four feature points are acquired, and the coordinates between the feature points are obtained. To obtain a coordinate transformation for transforming one image into the other image.

Next, this coordinate transformation is applied to the other feature points remaining in one image, and the degree of success of the coordinate transformation in the vicinity of the corresponding feature point in the other image is obtained.

Then, the random selection and the process for obtaining the success degree of coordinate transformation are repeatedly executed, and the coordinate transformation having the highest success degree is selected.

Then, feature point pairs whose coordinates are not converted to the vicinity by the coordinate conversion are removed as illegal.

The coordinate transformation obtained in this process corresponds to a second mapping in which each pixel of the search image is associated with any pixel of any one of the plurality of photo images.

Note that a Delaunay triangulation can be constructed using the feature points of the search image and applied to the feature points of the matched image to obtain a second mapping with higher accuracy.

That is, among the feature point pairs, a triangle mesh is generated using Delaunay triangulation for the feature points included in the search image, and the triangular region surrounded by the feature points is parameterized.

If the obtained mesh phase is applied as it is to the match image, it can be constructed in a region where similar triangles match.

Using the three vertices t ₀ , t ₁ , t ₂ constituting the triangle T of the search image, the barycentric coordinates (x, y, z) for the point p in the triangle T are
p = x ・ t ₀ + y ・ t ₁ + z ・ t ₂
It can be defined uniquely.

When the triangle T (vertices t ₀ , t ₁ , t ₂ ) in the search image is associated with the triangle T ′ (vertices t ′ ₀ , t ′ ₁ , t ′ ₂ ) in the match image by the feature point pair, Point p in triangle T goes to point p 'in triangle T'
p '= x ・ t' ₀ + y ・ t ' ₁ + z ・ t' ₂
The coordinates are converted as follows.

Thus, by defining the second map based on the triangle surrounded by the feature points, a more accurate second map can be obtained.

When the second map is obtained, the computer calculates the coordinate value in the wall map by applying the second map and the first map to each pixel in the search image (step S304). As described above, the coordinate value obtained here can be a three-dimensional coordinate or a two-dimensional coordinate.

Finally, the computer outputs the position in the wall map of the coordinate value calculated for each pixel in the search image (step S305), and ends this process.

In addition, instead of outputting the calculated position for each pixel, if an inspector designates a desired pixel in the search image or gives a pixel in which a deformation estimated by image recognition is drawn The position in the wall map for the pixel may be calculated and output. In this case, since most of the process of applying and outputting the first and second mappings for all the pixels in the search image can be omitted, the process can be speeded up.

(Use of GPU)
In the above calculation, for example, in order to calculate the mapping destination p ′ of the point p, it is necessary to obtain a triangle T including the point p. It is not obvious which triangle the point p is contained in.

単純 The simple method of checking the inclusion relation for all triangles is inefficient because it takes a computation time proportional to the number of triangles.

方法 There is also a method of starting from an arbitrary vertex and tracing adjacent triangles in the direction close to p, but it is necessary to have the adjacency relationship in a graph structure.

Therefore, in the present embodiment, an OpenGL FBO is drawn using a GPU, and a simple and high-speed calculation of a pixel is robustly realized by referring to the result.

For example, for each point on the surface of all polygons included in the 3D polygon model, the 3D coordinates of each point and the 2D coordinates when each point is shown in the development view are RGB (Red (Green Blue) The converted value is assigned as a color.

Then, the three-dimensional polygon model is perspective-projected from the photographing position and photographing direction obtained as a result of SfM. Then, the rendering result is the same composition as the original photographic image, and the image of each pixel is drawn as a result of converting the coordinates in the wall map into colors. The first mapping can be easily represented by this image.

In addition, when the search image and the match image are divided into triangles, colors that do not overlap each other are given to the vertices of the triangles, and colors that are proportional to the vertex colors by the barycentric coordinates are given to the inside of the triangles. For example, a vertex color obtained by converting a two-dimensional coordinate value of the vertex into a color can be used.

すると Then, when this triangle is converted from one to the other by coordinate transformation, the vertex color or a linear sum color is drawn inside the triangle, and no color is drawn outside the triangle.

Thus, it becomes possible to easily determine whether the triangle is inside or outside depending on the presence or absence of the color.

OpenIn OpenGL, each RGB component of a color is expressed as a floating point number between 0 and 1. Therefore, the three-dimensional coordinate value may be converted into a color by normalizing each three-dimensional element to a value between 0 and 1 and assigning it to each element of R, G, and B. In the case of a two-dimensional coordinate value, it is easiest to use only two components of R, G, and B.

(Use of laser scanner)
For example, a laser scanner conventionally used for tunnel wall inspection as disclosed in Patent Document 1 can measure cracks on the order of 1 mm or more from an automobile traveling at a speed of 50 km / h. However, in order to detect finer cracks and steps on the wall surface in the tunnel and the presence or absence of moisture, it is necessary to use a laser scanner with higher definition and higher accuracy. For example, in order to detect cracks of about 0.2mm-0.3mm and steps of about 0.1mm from a distance of about 5m, the resolution with a horizontal resolution of 0.2mm or less and a depth (ranging) resolution of 0.1mm or less is required. Necessary.

Here, using a high-definition, high-precision laser scanner enables more detailed information to be acquired than when shooting moving images or still images. However, since detection using a high-definition and high-precision laser scanner takes time, it is practically difficult to inspect the entire wall surface in the tunnel using a high-definition and high-precision laser scanner. .

かかわらず Regardless of the resolution, it is difficult to obtain the shooting position, shooting direction, and position of the shooting target in the measurement by the laser scanner, as in the inspection for shooting a moving image or a still image.

Therefore, if the above embodiment is applied when measuring the deformation range and its surroundings that need to be continuously monitored as a measurement range and measuring the measurement range with a laser scanner, the range measured by the laser scanner becomes a wall surface. You can get where in the map. This will be described below.

In the Enomoto method, when the retrieval image is photographed, the camera captures the wall surface in the tunnel continuously while maintaining the photographing position and the photographing direction within a predetermined error range.

During this continuous shooting, the laser beam from the laser scanner is applied to the measurement range from the middle.

In other words, the photograph obtained in the first half of the continuous shooting shows that the wall surface is photographed as it is, and the photograph obtained in the second half shows that a part of the wall surface is brightened with laser light.

Therefore, the photograph obtained in the first half of the continuous shooting is used as the search image. The photograph obtained in the second half of the continuous shooting represents a scan area by the laser scanner, and is hereinafter referred to as a scan image. The scan image can be almost overlapped with the search image, and the correspondence between the pixels can be easily determined.

Note that even if the shooting position or shooting direction shift is large to some extent, the correspondence between each pixel of the search image and the scanned image can be obtained by using the feature point extraction and matching technique in SfM. This is called the third map.

In other words, the position in the wall map can be easily obtained by applying the third map, the second map, and the first map to each point of the measurement range in the scan image.

It should be noted that by taking the search image first and then the scan image, it is possible to avoid the effects of afterimages on the camera and shorten the continuous shooting time.

(Experiment 1)
Below, the result of the experiment which applied this embodiment with respect to a part of simulated tunnel in the construction technology research institute in Fuji City, Shizuoka Prefecture is demonstrated.

DMPanasonic DMC-GF6 was used as a camera, and 250 photographic images were taken to cover the entire wall surface in the tunnel. The resolution of each photographic image is 1148 x 862 pixels. A three-dimensional model was generated from this photographic image.

検索 Also, using the same camera, a search image with a resolution of 4592 x 3598 pixels was taken. The search image is a close-up image on the wall surface, and the resolution is higher than that of the photographic image taken by initialization, but the field of view taken is narrow.

FIG. 15 is a drawing-substituting photograph that represents an example of a search image in 256 gray scales in Experiment 1. FIG. 16 is a drawing-substituting photograph that represents an example of a retrieval image in Experiment 1 in monochrome with two gradations. In the search images shown in these figures, wide T-shaped cracks are drawn with emphasis on the side.

FIG. 17 is a drawing-substituting photograph that represents, in Experiment 1, a photographic image that matches the search image in a gray scale of 256 gradations. FIG. 18 is a drawing-substituting photograph that represents, in Experiment 1, a photographic image that matches the search image in two-tone monochrome. This is a photographic image (match image) that matches the search image among the photographic images used at the time of initialization. In the match image, there is a wide T-shaped crack next to it, which is drawn with emphasis. Further, the pattern of the wall arranged around the T-shaped crack matches the search image and the match image.

FIG. 19 is a drawing-substituting photograph showing the three-dimensional polygon model for which the search image is searched in Experiment 1 with a gray scale of 256 gradations. FIG. 20 is a drawing-substituting photograph that represents the three-dimensional polygon model for which the search image is searched in Experiment 1 in two gradations. In this 3D polygon model, the photographic image referenced at the time of initialization is pasted as a texture, and the position with the same pattern as the search image is identified as the search result between the U shape of the tunnel edge. The T-shaped crack is mapped.

通り As shown in these figures, it can be confirmed that the crack information in the search image was identified at almost the same position of the corresponding 3D polygon model. Although the accuracy of the position depends on the data accuracy of SfM, since the relative positional relationship is preserved, the position of the search image on the tunnel wall surface can be appropriately identified.

(Experiment 2)
In the following, we explain the results of an experiment using OpenMVG for SfM, libigl for parameterization, and SSD for curved surface reconstruction for the simulated tunnel in the laboratory (total length 80m).

572SfM reconstruction used 572 images. Each image had a size of 1124 × 750 pixels and was photographed with a Nikon (registered trademark) camera D5500. It took about an hour to take all the photos.

For image search, first, cracks were taken with a camera DMC-GF6 manufactured by Panasonic (registered trademark), and five photographs of 1148 × 862 pixels were obtained. FIG. 21A is a drawing-substituting photograph showing an example of a cracked photographic image A to be searched in Experiment 2 in a gray scale of 256 gradations. FIG. 21B is a drawing-substituting photograph showing an example of a cracked photographic image B to be searched in Experiment 2 in a gray scale of 256 gradations. FIG. 21C is a drawing-substituting photograph that represents an example of a cracked photographic image C to be searched in Experiment 2 in a gray scale of 256 gradations. FIG. 21D is a drawing-substituting photograph showing an example of a cracked photographic image D to be searched in Experiment 2 in a gray scale of 256 gradations. FIG. 21E is a drawing-substituting photograph showing an example of a cracked photographic image E to be searched in Experiment 2 in a gray scale of 256 gradations. FIG. 22A is a drawing-substituting photograph showing an example of a cracked photographic image A to be searched in Experiment 2 in monochrome of two gradations. FIG. 22B is a drawing-substituting photograph showing an example of a cracked photographic image B to be searched in Experiment 2 in monochrome of two gradations. FIG. 22C is a drawing-substituting photograph showing an example of a cracked photographic image C to be searched in Experiment 2 in monochrome of two gradations. FIG. 22D is a drawing-substituting photograph showing an example of a cracked photographic image D to be searched in Experiment 2 in monochrome of two gradations. FIG. 22E is a drawing-substituting photograph that represents an example of a cracked photographic image E to be searched in Experiment 2 in monochrome with two gradations. The photograph actually taken is a color image.

Next, search images were obtained by manually extracting cracks from each photo. FIG. 23A is a drawing-substituting photograph that represents a search image for the photograph image A in Experiment 2 in monochrome with two gradations. FIG. 23B is a drawing-substituting photograph that represents the search image for the photograph image B in Experiment 2 in monochrome with two gradations. FIG. 23C is a drawing-substituting photograph in which the search image for the photographic image C in Experiment 2 is represented in monochrome with two gradations. FIG. 23D is a drawing-substituting photograph that represents a search image corresponding to the photograph image D in Experiment 2 in monochrome with two gradations. FIG. 23E is a drawing-substituting photograph in which the search image for the photographic image E in Experiment 2 is represented in monochrome with two gradations. As shown in these figures, the search image is a monochrome image.

写真 Then, photographic images matching the search image were searched from the 572 photos by the above method. FIG. 25A is a drawing-substituting photograph in which the photographic image A that matches the search image A in Experiment 2 is represented by a gray scale of 256 gradations. FIG. 25B is a drawing-substituting photograph in which the photographic image B matched with the search image B in Experiment 2 is represented by a gray scale of 256 gradations. FIG. 25C is a drawing-substituting photograph in which the photographic image C matching the search image C in Experiment 2 is represented by a gray scale of 256 gradations. FIG. 25D is a drawing-substituting photograph in which the photographic image D matched with the search image D in Experiment 2 is represented by 256 gray scales. FIG. 25E is a drawing-substituting photograph in which the photographic image E that matches the search image E in Experiment 2 is represented by a gray scale of 256 gradations. FIG. 24A is a drawing-substituting photograph in which the photographic image A that matches the search image A in Experiment 2 is represented in two-tone monochrome. FIG. 24B is a drawing-substituting photograph in which the photographic image B that matches the search image B in Experiment 2 is represented in two-tone monochrome. FIG. 24C is a drawing-substituting photograph in which the photographic image C that matches the search image C in Experiment 2 is represented in two-tone monochrome. FIG. 24D is a drawing-substituting photograph that represents the photographic image D that matches the search image D in Experiment 2 in monochrome with two gradations. FIG. 24E is a drawing-substituting photograph in which the photographic image E that matches the search image E in Experiment 2 is represented in two-tone monochrome. The photograph actually taken is a color image.

In this experiment, the search method described above was run on the computer MacBook (registered trademark) Pro (2016, Core i7 2.9GHz, 16GB RAM). FIG. 27 is a drawing-substituting photograph that expands the search result of the search image A-E in the development of the wall surface in the tunnel in Experiment 2 and expresses it in two-tone monochrome. FIG. 26 is a drawing-substituting photograph that is obtained by enlarging the result of searching for the search image A-E in the development of the wall surface in the tunnel in Experiment 2 and expressing it in 256 gray scales. These drawings are developed views of the inner surface of the tunnel, and show a state in which the position of a photographic image as a search result is surrounded by an ellipse and a crack is enlarged beside it. The search took 6.7 seconds to load the data, and the search itself took an average of 16.7 seconds. Therefore, it took 23.4 seconds on average to search for one search image, and it was found that the search can be performed in a practical time.

It took about 3 hours to reconstruct the 3D model. FIG. 29 is a drawing-substituting photograph in which the three-dimensional polygon model for which the search images A to E are searched in Experiment 2 is represented in two-tone monochrome. FIG. 28 is a drawing-substituting photograph showing the three-dimensional polygon model in which the search images A to E are searched in Experiment 2 in a gray scale of 256 gradations. Since each point in these drawings is associated with each point in the development view, it is possible to confirm the state of the cracked portion in a three-dimensional manner. It can also be seen that the reconstruction of the 3D model is done in a reasonable time.

(Summary)
As described above, the identification device according to the present embodiment is
A first acquisition unit for acquiring a plurality of photographic images of the walls in the tunnel;
A first calculation unit for calculating a position of each feature point and a local feature amount of each of the plurality of photographic images;
A three-dimensional polygon model of the wall surface is constructed with reference to the calculated feature point position and local feature amount for each of the plurality of photographic images, and a wall surface map based on the constructed three-dimensional polygon model is generated. Construction Department,
A first mapping unit that associates each pixel of the plurality of photographic images with a position in the wall surface map;
A second acquisition unit for acquiring a search image in which a wall surface in the tunnel is newly photographed;
A second calculator for calculating the position of the feature point of the acquired search image and the local feature amount;
By comparing the calculated feature point position and local feature amount for the search image with the calculated feature point position and local feature amount for each of the plurality of photographic images, A second mapping unit that associates each pixel with one of the pixels of the plurality of photographic images;
An output unit that outputs a position in the wall map in which any pixel of the plurality of photographic images associated with each pixel of the search image is associated;

In the identification apparatus according to the present embodiment,
The second acquisition unit is a scan image that is continuously shot while the shooting position and the shooting direction are maintained within a predetermined error range following the shooting of the search image, and the scan area in the wall surface is scanned with a laser scanner. Acquire a scanned image of the scan taken by
The second mapping unit associates each pixel of the scan image with any pixel of the search image by comparing the scan image with the search image.
The output unit associates any pixel of the plurality of photographic images associated with any pixel of the search image associated with each pixel in the scan area captured in the scan image. The position in the wall surface map can be output.

In the identification apparatus according to the present embodiment,
The wall surface map can be configured to be a three-dimensional map that represents a three-dimensional coordinate value of the wall surface by the three-dimensional polygon model.

In the identification apparatus according to the present embodiment,
The wall surface map can be configured to be a two-dimensional map that represents a two-dimensional coordinate value of the wall surface by a developed view of the three-dimensional polygon model.

In the identification apparatus according to the present embodiment,
The first mapping unit includes
Converting the coordinate value into color information;
Assign the converted color information as the color of the point associated with the coordinate value of the three-dimensional polygon model,
Rendering the three-dimensional polygon model to which the color is assigned from the shooting position and shooting direction at which each of the plurality of photographic images was taken, and generating a corresponding image having the same size as each of the plurality of photographic images ,
By inversely transforming the color drawn on each pixel in the generated corresponding image into coordinate values,
Each pixel of the plurality of photographic images can be configured to correspond to a position in the wall surface map.

In the identification apparatus according to the present embodiment,
The rendering can be configured to be executed by a GPU (Graphics Processing Unit).

In the identification method according to the present embodiment, the identification device is
Acquire multiple photographic images of the walls in the tunnel,
Calculating the position of each feature point and local feature amount of each of the plurality of photographic images;
A three-dimensional polygon model of the wall surface is constructed with reference to the calculated feature point positions and local feature amounts for each of the plurality of photographic images, and a wall map based on the constructed three-dimensional polygon model is generated. ,
Associating each pixel of the plurality of photographic images with a position in the wall map,
Obtain a search image in which the wall surface in the tunnel is newly photographed,
Calculating the position of the feature point of the acquired search image and the local feature amount;
By comparing the calculated feature point position and local feature amount for the search image with the calculated feature point position and local feature amount for each of the plurality of photographic images, Associating each pixel with one of the pixels of the plurality of photographic images,
A position in the wall surface map in which any pixel of the plurality of photographic images associated with each pixel of the search image is associated is output.

The program according to this embodiment is
The first computer,
A first acquisition unit for acquiring a plurality of photographic images of the walls in the tunnel;
A first calculation unit for calculating a position of each feature point and a local feature amount of each of the plurality of photographic images;
A three-dimensional polygon model of the wall surface is constructed with reference to the calculated feature point position and local feature amount for each of the plurality of photographic images, and a wall surface map based on the constructed three-dimensional polygon model is generated. Construction Department,
A first program that functions as a first mapping unit that associates each pixel of the plurality of photographic images with a position in the wall surface map;
A second computer or the first computer,
A second acquisition unit for acquiring a search image in which a wall surface in the tunnel is newly photographed;
A second calculator for calculating the position of the feature point of the acquired search image and the local feature amount;
By comparing the calculated feature point position and local feature amount for the search image with the calculated feature point position and local feature amount for each of the plurality of photographic images, A second mapping unit that associates each pixel with one of the pixels of the plurality of photographic images;
A second program that functions as an output unit that outputs a position in the wall map in which any pixel of the plurality of photographic images associated with each pixel of the search image is associated;
Is provided.

The program can be recorded and distributed and sold on a non-transitory computer-readable information recording medium. It can also be distributed and sold via a temporary transmission medium such as a computer communication network.

Various embodiments and modifications can be made to the present invention without departing from the broad spirit and scope of the present invention. The above-described embodiments are for explaining the present invention and do not limit the scope of the present invention. In other words, the scope of the present invention is shown not by the embodiments but by the claims. Various modifications within the scope of the claims and within the scope of the equivalent invention are considered to be within the scope of the present invention.
In this application, we shall claim priority based on patent application Japanese Patent Application No. 2017-033771 filed on February 24, 2017 (Friday) to Japan. The content of the basic application will be incorporated into this application as far as it permits.

101 identification device 111 first acquisition unit 112 first calculation unit 113 construction unit 114 first mapping unit 121 second acquisition unit 122 second calculation unit 124 second mapping unit 125 output unit

Claims

A first acquisition unit for acquiring a plurality of photographic images of the walls in the tunnel;
A first calculation unit for calculating a position of each feature point and a local feature amount of each of the plurality of photographic images;
A three-dimensional polygon model of the wall surface is constructed with reference to the calculated feature point position and local feature amount for each of the plurality of photographic images, and a wall surface map based on the constructed three-dimensional polygon model is generated. Construction Department,
A first mapping unit that associates each pixel of the plurality of photographic images with a position in the wall surface map;
A second acquisition unit for acquiring a search image in which a wall surface in the tunnel is newly photographed;
A second calculator for calculating the position of the feature point of the acquired search image and the local feature amount;
By comparing the calculated feature point position and local feature amount for the search image with the calculated feature point position and local feature amount for each of the plurality of photographic images, A second mapping unit that associates each pixel with one of the pixels of the plurality of photographic images;
An identification apparatus comprising: an output unit that outputs a position in the wall map in which any pixel of the plurality of photographic images associated with each pixel of the search image is associated.
The second acquisition unit is a scan image that is continuously shot while the shooting position and the shooting direction are maintained within a predetermined error range following the shooting of the search image, and the scan area in the wall surface is scanned with a laser scanner. Acquire a scanned image of the scan taken by
The second mapping unit associates each pixel of the scan image with any pixel of the search image by comparing the scan image with the search image.
The output unit associates any pixel of the plurality of photographic images associated with any pixel of the search image associated with each pixel in the scan area captured in the scan image. 2. The identification apparatus according to claim 1, wherein a position in the wall surface map is output.
2. The identification device according to claim 1, wherein the wall map is a three-dimensional map that represents a three-dimensional coordinate value of the wall surface by the three-dimensional polygon model.
2. The identification device according to claim 1, wherein the wall surface map is a two-dimensional map that represents a two-dimensional coordinate value of the wall surface by a developed view of the three-dimensional polygon model.
The first mapping unit includes
Converting the coordinate value into color information;
Assign the converted color information as the color of the point associated with the coordinate value of the three-dimensional polygon model,
Rendering the three-dimensional polygon model to which the color is assigned from the shooting position and shooting direction at which each of the plurality of photographic images was taken, and generating a corresponding image having the same size as each of the plurality of photographic images ,
By inversely transforming the color drawn on each pixel in the generated corresponding image into coordinate values,
5. The identification apparatus according to claim 3, wherein each pixel of the plurality of photographic images is associated with a position in the wall map.
6. The identification apparatus according to claim 5, wherein the rendering is executed by a GPU (Graphics Processing Unit).
The identification device
Acquire multiple photographic images of the walls in the tunnel,
Calculating the position of each feature point and local feature amount of each of the plurality of photographic images;
A three-dimensional polygon model of the wall surface is constructed with reference to the calculated feature point positions and local feature amounts for each of the plurality of photographic images, and a wall map based on the constructed three-dimensional polygon model is generated. ,
Associating each pixel of the plurality of photographic images with a position in the wall map,
Obtain a search image in which the wall surface in the tunnel is newly photographed,
Calculating the position of the feature point of the acquired search image and the local feature amount;
By comparing the calculated feature point position and local feature amount for the search image with the calculated feature point position and local feature amount for each of the plurality of photographic images, Associating each pixel with one of the pixels of the plurality of photographic images,
An identification method, comprising: outputting a position in the wall map in which any pixel of the plurality of photographic images associated with each pixel of the search image is associated.
The first computer,
A first acquisition unit for acquiring a plurality of photographic images of the walls in the tunnel;
A first calculation unit for calculating a position of each feature point and a local feature amount of each of the plurality of photographic images;
A three-dimensional polygon model of the wall surface is constructed with reference to the calculated feature point position and local feature amount for each of the plurality of photographic images, and a wall surface map based on the constructed three-dimensional polygon model is generated. Construction Department,
A first program that functions as a first mapping unit that associates each pixel of the plurality of photographic images with a position in the wall surface map;
A second computer or the first computer,
A second acquisition unit for acquiring a search image in which a wall surface in the tunnel is newly photographed;
A second calculator for calculating the position of the feature point of the acquired search image and the local feature amount;
By comparing the calculated feature point position and local feature amount for the search image with the calculated feature point position and local feature amount for each of the plurality of photographic images, A second mapping unit that associates each pixel with one of the pixels of the plurality of photographic images;
A second program that functions as an output unit that outputs a position in the wall map in which any pixel of the plurality of photographic images associated with each pixel of the search image is associated;
A program comprising: