JP2020008802A - Three-dimensional map generation device and three-dimensional map generation method - Google Patents

Abstract

To generate a three-dimensional map for permitting detection of a new change on a satellite image by reflecting the most recent situation of the side of a three-dimensional structure by the satellite image.SOLUTION: A three-dimensional map generation device for generating a three-dimensional map from satellite image data generates a three-dimensional point group from the satellite image data, changes a surface of a shape expressed by the three-dimensional point group into meshes, and generates mesh data for expressing it, and selects an image to be pasted on the basis of photographing conditions of a satellite image and pastes it to a mesh surface for meshes expressed by the mesh data. An image having a new photographing date/time of a satellite image is selected for an image to be pasted for meshes. A side vertical to the ground of an object as the side of a three-dimensional shape expressed by the three-dimensional point group is detected and an image is pasted to the side.SELECTED DRAWING: Figure 7

Description

  The present invention relates to a three-dimensional map generation device and a three-dimensional map generation method, and in particular, to generating a three-dimensional map by using a plurality of satellite images, in generating a three-dimensional map that reflects the latest situation of an imaging target. The present invention relates to a three-dimensional map generation device and a three-dimensional map generation method suitable for the present invention.

  An artificial satellite image (also simply referred to as “satellite image” in the present specification) is an image obtained by photographing an object on the ground from an artificial satellite around the earth. With the globalization of information due to the spread of information, it is expected to be used in various fields such as disaster prevention, mapping, academic research, and business use. In satellite images, the ability to capture a wide area with a single image capture detects changes in ground coverage in the same area and the appearance of features that did not exist previously in new images. Can be. On the other hand, in the use of satellite images, an analysis platform capable of quickly grasping the situation over a wide area (object extraction / change extraction) is required with the increase in large-scale disasters and the tense situation in Asia. Therefore, many satellite images are needed to monitor changes in a wide area over a long period of time and to obtain meaningful information.

  Conventionally, there have been Landsat and MODIS as artificial satellites that provide images that can be downloaded free of charge via the Internet. However, these images have a problem of low resolution of 30 m and 250 m in a multispectral band. In recent years, with the advent of ultra-small satellites equipped with high-resolution sensors, the cost of satellite image acquisition has been reduced, and Digital Grove, a US company that provides high-resolution satellite images for the private sector, has released the GBDX platform. At the dawn of the use of high-resolution satellite image archives, the study of solutions based on a large number of high-resolution satellite images has begun.

  With the increase in the accuracy of such satellite images, it has become possible to detect detailed changes in the ground surface.However, when analyzing satellite images, the occurrence and location of invisible regions due to differences in elevation of structures on the ground surface Misalignment is a problem. These problems occur because the satellite image assumes that the height of the object to be photographed is the same as the ground surface. In the conventional low-quality satellite image, the error caused by the height of the target object was not a problem because it was smaller than the error of the satellite image, but with the increase in the accuracy of the satellite image, the error was relatively small. It can be said that the problem has become apparent due to the increase in size. In order to eliminate such an error caused by the height of the object, the error can be eliminated by acquiring altitude information of the object on the ground.

  As a technique for solving such a problem and creating a high-accuracy map from a photographed image of an artificial satellite, for example, there is a technique disclosed in Patent Document 1. The map creation method described in Patent Document 1 takes images of the ground by using satellites and airplanes, aerial photographs have extremely high resolution, and satellite photographs have extremely small distortions. To obtain an image without distortion.

JP 2000-276045 A

  When trying to grasp the shape of an object from a satellite image in the prior art, when detecting a change using the satellite image, if the target location is the side of the three-dimensional structure, does the image to be compared show the side? It is necessary to confirm whether or not there is any for each target location. Even if the target location is shown, it is necessary to check the latest change in order from a new satellite image in order to detect the latest change. When there are countless target locations, the burden on the reader is large.

  Patent Literature 1 described in the above-mentioned prior art discloses a technique of complementing a blank portion with a satellite photograph for a blank portion in which aerial photograph distortion is complemented with a satellite photograph (paragraph number 0042, FIG. d)). However, the map creation method described in Patent Literature 1 targets a two-dimensional map, and suggests how to grasp and photograph the side surface of a three-dimensional structure such as a building to create a three-dimensional map. It has not been. Also, no consideration is given to the fact that the latest image of the satellite image is reflected on the three-dimensional map.

  SUMMARY OF THE INVENTION An object of the present invention is to provide a three-dimensional map generating apparatus and a three-dimensional map generating apparatus capable of generating a three-dimensional map capable of detecting a new change on a satellite image by reflecting the latest situation of the side surface of the three-dimensional structure by the satellite image An object of the present invention is to provide an original map generation method.

  The configuration of the three-dimensional map generation device of the present invention is preferably a three-dimensional map generation device that generates a three-dimensional map from satellite image data, and generates a three-dimensional point group from the satellite image data. Meshes the surface of the shape represented by, generates mesh data representing it, selects an image to be pasted on the mesh represented by the mesh data based on the shooting conditions of the satellite image, and It is intended to be pasted.

  According to the present invention, a three-dimensional map generation device and a tertiary map capable of generating a three-dimensional map capable of detecting a new change on a satellite image by reflecting the latest state of the side surface of the three-dimensional structure based on the satellite image An original map generation method can be provided.

It is a functional block diagram of a three-dimensional map generation apparatus. FIG. 3 is a functional configuration diagram of a three-dimensional point cloud generation unit. FIG. 2 is a hardware / software configuration diagram of the three-dimensional map generation device. It is a figure showing an example of satellite image data. FIG. 3 is a diagram illustrating an example of three-dimensional data. FIG. 4 is a diagram illustrating an example of mesh data. It is a flowchart which shows a three-dimensional map generation process. FIG. 3 is a diagram illustrating an outline of three-dimensional map generation. FIG. 3 is a diagram illustrating projection conversion. It is a figure explaining the detection procedure of the side of a solid (the 1). It is a figure explaining the detection procedure of the side of a solid (No. 2). FIG. 9 is a diagram illustrating an effect of the three-dimensional map generation device according to the embodiment as compared with the related art.

  An embodiment according to the present invention will be described below with reference to FIGS.

First, the configuration of the three-dimensional map generation device will be described with reference to FIGS.
As shown in FIG. 1, the three-dimensional map generation device 100 according to the present embodiment has a satellite image selection unit 101, a satellite image distortion correction unit 102, a three-dimensional point group generation unit 103, a three-dimensional point group It is composed of a surface sticking unit 104, a result output unit 105, and a storage unit 200.

  The satellite image selection unit 101 is a functional unit that selects a satellite image required to generate a three-dimensional map based on an input satellite image. The satellite image distortion correction unit 102 is a functional unit that corrects distortion of an input satellite image. The three-dimensional point cloud generation unit 103 is a functional unit that generates a three-dimensional point cloud for generating a three-dimensional map. The three-dimensional point group surface pasting unit 104 is a functional unit that pastes a three-dimensional side surface represented by a three-dimensional point cloud as an image. The result output unit 105 is a functional unit that outputs the generated three-dimensional map. The storage unit 200 is a functional unit that stores data necessary for generating a three-dimensional map.

  The storage unit 200 stores satellite image data 201, three-dimensional point cloud data 202, mesh data 203, and input satellite image data 204.

The satellite image data 201 is data relating to an image taken from a satellite. The three-dimensional point cloud data 202 is three-dimensional coordinate data for expressing an imaging target in order to generate a three-dimensional map. The mesh data 203 is data that captures the three-dimensional point cloud data 202 as a group and expresses it as a mesh. The input satellite image data 204 is satellite image data serving as a source for narrowing down an object when a three-dimensional map is generated from the satellite image data 201.
Each data structure will be described later in detail.

Next, the detailed structure of the three-dimensional point cloud surface attaching portion will be described with reference to FIG.
As shown in FIG. 2, the three-dimensional point cloud surface pasting unit 104 includes a three-dimensional point cloud meshing unit 401, a mesh pasting satellite image selecting unit 402, a satellite image mesh region extracting unit 403, and a satellite image mesh pasting unit 406. Become.

  The three-dimensional point group meshing unit 401 is a functional unit that forms a mesh from a three-dimensional point group. The mesh pasting satellite image selection unit 402 is a functional unit that selects a satellite image to be pasted on a mesh. The satellite image mesh region extraction unit 403 is a functional unit that cuts out a satellite image selected to be pasted on a mesh. The satellite image mesh pasting unit 406 is a functional unit for pasting a portion extracted from the selected satellite image to a mesh.

  The satellite image mesh area extraction unit 403 includes a satellite image projection conversion detection unit 404 and a side position detection unit 405.

  The satellite image projection conversion detection unit 404 is a functional unit that detects a satellite image to be extracted by a projection conversion technique (described later). The side surface position detection unit 405 is a functional unit that detects, in particular, a three-dimensional side surface for pasting a satellite image (detailed algorithm will be described later).

Next, the hardware and software configuration of the three-dimensional map generation device will be described with reference to FIG.
The hardware configuration of the three-dimensional map generation device 100 is realized by, for example, an information processing device such as a general personal computer as shown in FIG.
The three-dimensional map generation device 100 has a form in which a CPU (Central Processing Unit) 502, a main storage device 504, a network I / F 506, a display I / F 508, an input / output I / F 510, and an auxiliary storage I / F 512 are connected by a bus. It has become.

  The CPU 502 controls each unit of the three-dimensional map generation device 100, loads necessary programs into the main storage device 504, and executes them. The main storage device 504 is generally configured by a volatile memory such as a RAM, and stores a program executed by the CPU 502 and data to be referred to. The network I / F 506 is an interface for connecting to a network. The display I / F 508 is an interface for connecting a display device 520 such as an LCD (Liquid Crystal Display). The input / output I / F 510 is an interface for connecting an input / output device. In the example of FIG. 3, a keyboard 530 and a mouse 532 as a pointing device are connected.

  The auxiliary storage I / F 515 is an interface for connecting an auxiliary storage device such as a hard disk drive (HDD) 550 or a solid state drive (SSD).

  The HDD 550 has a large storage capacity, and stores a program for executing the present embodiment. In the three-dimensional map generation device 100, a satellite image selection program 601, a satellite image distortion correction program 602, a three-dimensional point cloud generation program 603, a three-dimensional point cloud surface pasting program 604, and a result output program 605 are installed. The satellite image selection program 601, the satellite image distortion correction program 602, the three-dimensional point cloud generation program 603, the three-dimensional point cloud surface pasting program 604, and the result output program 605 are respectively a satellite image selection unit 101, a satellite image distortion correction unit 102, This is a program for implementing the functions of the three-dimensional point cloud generation unit 103, the three-dimensional point cloud surface pasting unit 104, and the result output unit 105.

  The HDD 550 stores satellite image data 201, three-dimensional point cloud data 202, mesh data 203, and input satellite image data 204.

Next, a data structure used in the three-dimensional map generation device will be described with reference to FIGS.
As shown in FIG. 4, the satellite image data 201 includes satellite image ID, shooting position, shooting direction, focal length, resolution, satellite position, shooting time, sensor angle, and band information.

  The satellite image ID is an ID for uniquely identifying a captured image. The band information describes the length of the wavelength of light acquired in each image, or a calculation formula using the acquired area and the value of the area. RGB images are often used as satellite images, but sensors that observe various frequencies such as infrared light can be used, and data formats other than general RGB images may be used. it can. Although the direction of the sun can be calculated based on the shooting time information, it may be described in the satellite image data in order to speed up the calculation.

  The three-dimensional point cloud data 202 is data representing a three-dimensional position on the satellite image, and includes three-dimensional coordinate ID, satellite image ID, and three-dimensional coordinate items as shown in FIG. The three-dimensional coordinates may be represented by orthogonal coordinates or polar coordinates.

  The mesh data 203 is data that represents the three-dimensional data as a mesh (mesh), and includes items of a mesh ID, a mesh vertex, and a normal vector. The mesh vertices store n (n is an integer of 1 or more) three-dimensional coordinate vertices. The normal vector is a three-dimensional representation of the normal vector of the mesh surface.

  The input satellite image data 204 is data on which a three-dimensional map is to be generated, and has the same data structure as the satellite image data 201. The user can determine which region the 3D map generates the 3D map at which date and time at which date and time, but in the present embodiment, in this embodiment, the 3D map at which date and time in which region A description will be given assuming that a reference for generating a map is given by the input satellite image data 204.

  Next, a three-dimensional map generation process performed by the three-dimensional map generation device will be described with reference to FIGS.

Hereinafter, the three-dimensional map generation processing of the three-dimensional map generation device according to the present embodiment will be described with reference to the flowchart illustrated in FIG.
FIG. 8 shows an outline of the three-dimensional map generation processing. First, a three-dimensional point group is generated from a plurality of images (FIGS. 8A to 8B, S201 to S203). Next, the surface of the shape constituted by the three-dimensional point group is meshed (FIG. 8B → (c), S204). Then, an image is cut out from a new satellite image for each of the meshes and attached to the side surface (FIGS. 8C to 8D, S205 to S207).

Hereinafter, the details will be described.
First, the input satellite image data 204 is input, and satellite image data corresponding thereto is acquired (S201). The input satellite image data is a satellite image used as a reference for generating a three-dimensional map, as described above.

  Here, the satellite image selecting unit 101 acquires the shooting date of the input input satellite image data 204, the angle of the sensor at the time of shooting, and the satellite position at the time of shooting, and obtains the angle of the sensor at the time of shooting, Calculate the shooting range by using the satellite position. Regarding the imaging conversion, by using the projection conversion, it is possible to calculate the latitude and longitude of the range where the satellite image is being captured. At the time of projection conversion, elevation data may be used. Then, a satellite image group for generating a three-dimensional map that overlaps with the shooting range and satisfies that the shooting date is before the shooting date of the input satellite image is extracted from the satellite image data 201.

  Next, the distortion of the satellite image is corrected (S202). In the case of a satellite image, it often takes a long time to capture an image, and the image is often distorted because the image is far from the subject. Therefore, a known object at a position serving as a marker is set on the satellite image to correct the distortion. In detecting an object, automatic detection or manual detection may be used. Further, depending on the method of obtaining the satellite image, some satellite images have already been corrected for distortion, so that the processing S202 can be omitted.

Next, a three-dimensional point group is generated by stereo vision (S203). The process of generating a three-dimensional point group by stereo vision is based on existing SfM (Structure for Motion) and MVS (Multi-View Stereo) techniques. This processing performs the following processing, which is an outline.
(1) Automatic extraction of feature points (2) Extraction of camera position and orientation and three-dimensional coordinates of feature points (3) Generation of three-dimensional point group by multi-view image measurement

  The generated three-dimensional point cloud is recorded in the three-dimensional point cloud data 202.

  The MVS is a method of providing a parallax image for realizing a stereoscopic three-dimensional video in the step (3), and is a method of restoring a three-dimensional shape based on matching between stereo images.

  The three-dimensional point cloud generation unit 103 acquires the sensor angle at the time of shooting each satellite image and the satellite position at the time of shooting from the satellite image data 201, and obtains the camera positions of two or more satellite images forming a stereo pair by MVS. It is adopted as the initial value of the posture. As a pre-process for performing the MVS, a relative relationship between the camera positions of the satellite images is acquired by SfM. In SfM, the correspondence between images is calculated by feature point matching between satellite images. In addition, in order to increase the accuracy of the process of calculating the correspondence between the images, the database of the optical performance of the camera is referred from the angle of the sensor at the time of shooting and the satellite position at the time of shooting, and information on the focal length, image Acquire camera internal parameters such as distortion information. Using these information and known GCP (Google Cloud Platform) information, correction of the position and orientation of the camera between a plurality of satellite images or correction of internal parameters is performed. As described above, it is preferable to improve the estimation accuracy of the position and orientation information of the camera serving as the base point of the stereo pair by using information having higher reliability than the feature points. Here, since reliable GCP information improves the generation accuracy of the three-dimensional shape, it is desirable to provide a mechanism for storing GCP information newly input by the user.

  Of course, in generating a three-dimensional map, it is not necessary to make it three-dimensional using only satellite images, and it is possible to use a model that adds map information to a three-dimensional thing using an image taken near a ground such as a drone. Alternatively, a three-dimensional point group acquired by a laser sensor may be used together. It is assumed that the positional accuracy of the three-dimensional shape acquired using a satellite image other than these is approximately one to two digits higher than the three-dimensional shape created using only the satellite image. Therefore, the calibration of the camera parameters of the satellite image may be performed using these pieces of position information. The camera parameters are parameters possessed by the camera regarding information on position and orientation, information on distortion, focal length, and the like. For example, when performing bundle adjustment, it is preferable to fix these three-dimensional point groups as known information and then search for a camera parameter that reduces a reprojection error for each satellite image. With the MVS using the information of the camera parameters calibrated in this way, it is expected that the position accuracy of the three-dimensional point group based on the satellite image is improved.

  Next, a meshing process is performed on the surface constituted by the three-dimensional point group (S204). Here, the meshing process described above refers to a process of filling the surface of a three-dimensional point group with a polygon. This mesh processing of the three-dimensional point group is performed in the three-dimensional point group surface attaching unit 104.

  First, the three-dimensional point cloud meshing unit 401 meshes the three-dimensional point cloud stored in the three-dimensional point cloud data 202. For the meshing algorithm, various methods such as Delaunay's triangulation method have been proposed, and these can be used. The calculated mesh is stored in the mesh data 203. What is stored are the vertices of the polygons constituting the mesh and the outward normal vectors of the polygons. The normal vector is a vector that is perpendicular to the plane and that is perpendicular to all the straight lines on the plane.

  Next, the satellite image mesh area extraction unit 403 extracts a satellite image that can be pasted (S205). A satellite image showing a mesh is extracted using the sensor angle at the time of shooting of the mesh data 203 and the satellite image data 201 and the satellite position at the time of shooting. Even if you are in the shooting range, it will not be reflected if there is an obstruction in front. However, at this time, if the normal vector of the mesh and the shooting angle of the satellite are significantly different, even if the shooting is successful, the mesh does not have to be extracted according to the angle difference because the resolution of the mesh is low.

  Next, a satellite image that can be pasted is determined (S206), and the image is pasted with reference to the region to be pasted by projection conversion (FIG. 9) (S207). Here, the newest (the latest in time) satellite image among the extracted satellite images is pasted. However, it is also possible to select a scoring and out-of-date image depending on the weather or shooting time zone. For example, some satellite images are accompanied by cloud position information on the satellite image, and the use of the latest image can be avoided using the cloud position information. The region of the image to be pasted is determined by the satellite image projection conversion detection unit 404, using the principle of the projection conversion shown in FIG. Can be determined. Although the shape is different but the mesh is flat, it can be pasted by using homography conversion. By performing this process for each mesh, a three-dimensional map can be created as a result.

  Finally, the result output unit 105 displays and outputs the information as three-dimensional map information on a display device or the like (S208). Further, the image viewed from the viewpoint may be displayed as a two-dimensional image using the angle of the sensor at the time of capturing the satellite image data and the satellite position at the time of capturing.

  In the above description, the satellite image mesh area extraction unit 403 extracts the satellite image of the area where the mesh is reflected, using the sensor angle at the time of shooting the mesh data 203 and the satellite image data 201 and the satellite position at the time of shooting. did.

  Here, a procedure for detecting a three-dimensional side surface of three-dimensional data will be described as one mode of extracting a mesh region with reference to FIGS. 10A and 10B.

  The side surface position detection unit 405 of the satellite image mesh region extraction unit 403 detects the mesh as a side surface when a line segment forming each mesh is perpendicular to the ground (FIG. 10A). Then, it is determined which region of the satellite image selected by the mesh-pasted satellite image selection unit 402 is to be extracted.

  Hereinafter, a procedure for detecting the three-dimensional side surface of the three-dimensional data for each step will be described with reference to FIG. 10B. First, edge detection of the mesh is performed (step 1). As the edge detection algorithm, various existing methods such as Hough transform have been proposed, and these may be used. By using the angle of the sensor at the time of shooting the satellite image and the satellite position at the time of shooting, the direction of the portion perpendicular to the ground on the satellite image can be calculated. Next, based on the direction of a line perpendicular to the ground determined by the position of the satellite image, a line segment constituting the side surface is detected (step 2). Here, the edge of the three-dimensional structure can be detected on the satellite image by detecting the edge close to the direction among the edges. Finally, a polygon including the vertical line segment is extracted (step 3). This makes it possible to detect an area used for pasting.

  In the above-described embodiment, an example has been described in which a three-dimensional map is generated from a satellite image. However, a two-dimensional image may be generated by specifying a three-dimensional map or a viewpoint. Also. The difference between the input satellite image and the comparison image may be displayed.

Next, the effect of the three-dimensional map generation of the present embodiment will be described with reference to FIG.
As shown in FIG. 8, in the three-dimensional map generation method (SfM, MVS) according to the related art, since the surface of a figure composed of a three-dimensional point group is statistically estimated, it is necessary to consider the latest state. Failed ((a-1)). In the present embodiment, the latest state can be considered in order to make the image as new as possible cut out from the satellite image on the surface of the figure ((b-1)).

Further, in the related art, the change of the immediately preceding imaging target can be detected by comparing the created three-dimensional map with the new satellite image ((a-2)). According to this, it is possible to detect only the latest change of the imaging target ((b-2)).
In the embodiment described above, a satellite image which is an example of an aerial image is described. However, an aerial image captured from another flying object such as an aircraft or a drone can be similarly applied.

Reference Signs List 100: three-dimensional map generation device 101: satellite image selection unit 102: satellite image distortion correction unit 103: three-dimensional point cloud generation unit 104: three-dimensional point cloud surface pasting unit 105: result output unit 201: satellite image data 202: tertiary Original point cloud data 203 Mesh data 204 Input satellite image data 401 Three-dimensional point cloud meshing section 402 Mesh pasting satellite image selecting section 403 Satellite image mesh area extracting section 404 Satellite image projection conversion detecting section 405 Side Position detection unit 406: Satellite image mesh attaching unit

Claims (5)

A three-dimensional map generation device that generates a three-dimensional map from satellite image data,
Generating a three-dimensional point cloud from the satellite image data;
By meshing the surface of the shape represented by the three-dimensional point group, mesh data representing the mesh is generated,
A three-dimensional map generation device, wherein an image to be pasted is selected for a mesh represented by the mesh data based on a photographing condition of the satellite image and pasted to the mesh.   2. The three-dimensional map generation device according to claim 1, wherein an image to be pasted on the mesh selects an image having a new shooting date and time of the satellite image. 3. The side surface of the three-dimensional shape represented by the three-dimensional point cloud, the side surface perpendicular to the ground of the object is detected from the shooting angle of the satellite image,
The three-dimensional map generation device according to claim 1, wherein the mesh on which the image is pasted is a detected side surface. A three-dimensional map generation method for generating a three-dimensional map from satellite image data,
Generating a three-dimensional point cloud from the satellite image data;
Meshing the surface of the shape represented by the three-dimensional point group, and generating mesh data representing the mesh;
For the mesh represented by the mesh data, based on the shooting conditions of the satellite image, the satellite image shooting date and time to select a new image to paste the image,
Attaching to the mesh,
Outputting the generated three-dimensional map. Furthermore, it has a step of detecting a side surface perpendicular to the ground of the object from the shooting angle of the satellite image, which is a side surface of the three-dimensional shape represented by the three-dimensional point cloud,
The three-dimensional map generation method according to claim 4, wherein the mesh to which the image is pasted is a detected side surface.

Images