JP6080640B2 - 3D point cloud analysis method - Google Patents

Description

  The present invention relates to a three-dimensional point cloud analysis method for analyzing a three-dimensional point cloud representing the shape of a plurality of features or the like.

In a three-dimensional map that three-dimensionally represents the shape of a feature, it is required to generate a three-dimensional model of the feature with high accuracy and a low processing load. As one of such techniques, use of three-dimensional restoration is being studied.
The three-dimensional restoration process is a process for obtaining a three-dimensional point group of photographed features based on the correspondence relationship between feature points between a plurality of images photographed from a plurality of points. If three-dimensional reconstruction is used, a plurality of images within the processing target range are taken from an aircraft or the like, and a three-dimensional point group representing the target range can be obtained by analyzing the images. If such a point cloud is obtained, it is possible to generate a three-dimensional model by analyzing it. SIFT (Scale-Invariant Feature Transform) or the like is used as a technique for determining the correspondence between feature points between images.

In order to generate a three-dimensional model, it is necessary to associate the coordinate system of the three-dimensional point group with the earth coordinate system.
Patent Document 1 discloses a technique of defining a horizontal plane based on a point group having a low height among three-dimensional point groups and further defining a vertical plane based on a point group close to the horizontal direction.
Patent Document 2 discloses a technique for defining a horizontal plane and a vertical plane by regarding a plane obtained by analyzing a three-dimensional point group as having a maximum area as a floor plane.

Japanese Patent Laid-Open No. 2003-256871 JP 2007-271408 A

  The three-dimensional coordinates given to each point in the three-dimensional point group are not necessarily obtained in the earth coordinate system, and the position coordinates in the three-dimensional space for analysis for analyzing the position of the point group from the image are It may only have been obtained. Even in other three-dimensional point groups, the position coordinates of the points are given in a coordinate system depending on the measurement method for obtaining the point group, and therefore, the position coordinates in the earth coordinate system may not be specified. When generating a three-dimensional model by analyzing a three-dimensional point cloud, it is necessary to represent the position and shape of each feature in the earth coordinate system, and it is necessary to specify the direction of gravity as the coordinate axis that serves as the basis. is there. An object of this invention is to provide the technique which identifies a gravity direction in the coordinate system to which a three-dimensional point group belongs based on this subject and analyzing a three-dimensional point group.

The present invention
A three-dimensional point cloud analysis method for analyzing a three-dimensional point cloud composed of a plurality of points representing the outer shape of a plurality of features by a computer,
The three-dimensional point group data constituting the three-dimensional point group stores position coordinates in a three-dimensional coordinate space set for analysis for each point for a plurality of points representing the outer shape of the feature,
In the three-dimensional point cloud analysis method, the computer executes the following steps:
(A) reading the three-dimensional point cloud data;
(B) identifying a side point group that is a set of points representing the side surface of the feature from the three-dimensional point group data;
(C) obtaining a normal vector of each side of the feature based on the side point group;
(D) Obtaining an outer product of two normal vectors selected from the normal vectors for selection of a plurality of sets, and determining a direction of gravity in the three-dimensional coordinate space based on statistical processing of the obtained plural outer products. A three-dimensional point cloud analysis method.
Here, the three-dimensional point group dealt with in the present invention can be a target obtained by various methods such as a technique using SIFT and SfM (Structure from Motion), laser measurement, and the like.

  The three-dimensional point group handled in the present invention represents a plurality of features. The shape of the side surface varies depending on the feature, and the accuracy of the coordinates of the three-dimensional point group obtained by the feature varies. In such a state, if the direction of gravity is determined by any local analysis, sufficient accuracy cannot be ensured. According to the present invention, since side surfaces of a plurality of features are used and statistical processing is performed, errors due to such variations can be suppressed, and the direction of gravity can be obtained with high accuracy.

In the present invention,
The three-dimensional point group is obtained for a region in which the distribution in the height direction of the feature is sufficiently smaller than the distribution in the horizontal direction due to the arrangement of the plurality of features, and the position for each point. Stores the normal vector of the surface where the point is assumed to exist along with the coordinates,
In the three-dimensional point cloud analysis method, the computer executes the following steps:
(E) setting a direction of a third principal component obtained by principal component analysis of the three-dimensional point group as a temporary gravity direction;
In the step (b), the side surface point group may be specified by excluding the point group in which the normal vector of each point is along the temporary gravity direction.
As a method for acquiring a three-dimensional point group from which a normal vector is obtained for each point, a technique using SIFT and SfM (Structure from Motion) can be cited.

According to the above aspect, the provisional gravity direction can be set by principal component analysis. Principal component analysis is an analysis method that analyzes the distribution of point groups and obtains the first principal component, the second principal component, and the third principal component in order from the direction of the largest variance. The first main component, the second main component, and the third main component are orthogonal components. According to this method, since the gravitational direction is calculated using the entire three-dimensional point group, the influence of errors included in each point can be suppressed and the calculation accuracy can be improved.
The distribution in the height direction in the three-dimensional point group used in the above aspect is sufficiently smaller than the distribution in the horizontal direction. This means that when the principal component analysis is performed, the first principal component and the second principal component appear in the horizontal direction, unlike the height direction. Therefore, if the third principal component of the principal component analysis is adopted, the gravity direction of the three-dimensional point group can be determined. However, since the gravitational direction obtained in this way includes an error, it represents the provisional gravitational direction.
It is relatively easy to acquire a three-dimensional point group whose distribution in the height direction and the horizontal direction satisfies the above-described conditions. Since the height of the feature is only a range up to about several hundreds of meters, the above-mentioned condition is naturally satisfied by obtaining a three-dimensional point group in a region having a radius of several times the feature height.

If the provisional gravity direction is determined in this way, the side surface point group can be accurately identified by excluding the point group along the gravity direction.
The specification of the side point group in the present invention is not necessarily limited to the method using the temporary gravity direction. For example, consider a case where each point constituting a three-dimensional point group is classified according to the direction of the normal vector and projected in the direction of the normal vector. Since the point having the normal vector in the upward direction is the upper surface or the ground surface of the feature, if this is projected, it should be possible to almost cover the region where the three-dimensional point group is obtained. Therefore, when projecting in the normal vector direction of each point, specify the side point group by specifying the direction in which the void is reduced as the top surface or the ground surface and excluding the corresponding point group in the projection result. can do.

In the present invention,
The step (c), that is, the step of obtaining the normal vector of each side of the feature is obtained by separating the side point group into each side of each feature and performing principal component analysis for each side. The principal component may be a normal vector of each side of the feature.
Since the point cloud constituting the side surface of the feature is distributed along the side surface, if the principal component analysis is performed, the first principal component and the second principal component appear in the direction along the side surface, and the third principal component is , The direction perpendicular to these directions, that is, the normal direction of the side surface. If the normal vector of each side surface is obtained by such a method, the influence of errors included in each point can be suppressed and the accuracy can be improved.
In the present invention, the normal vector of the side surface can be obtained by other methods. For example, you may obtain | require by the average of the normal vector of each point which comprises a side surface point group, etc. Further, a method may be used in which a plane passing through the center of these point groups is obtained on the basis of the side point group by the least error square method and the normal vector of this plane is obtained.

In the aspect of obtaining the normal vector of the side surface by principal component analysis as described above,
The step (d), that is, the step of determining the direction of gravity by statistical processing, uses the third eigenvalue obtained for the normal vector in the principal component analysis to calculate the outer product of the normal vectors in each set. A weight value representing accuracy may be calculated, and a weighted average of the outer product using the weight value may be obtained as the statistical processing.
The first to third eigenvalues in the principal component analysis are values representing the magnitude of dispersion in each principal component. When the principal component analysis is performed on the point group constituting the side surface, if the point group accurately constitutes the side surface, the third eigenvalue should be small because the variance in the third principal component direction is small. That is, according to the third eigenvalue, it is possible to set a weight value representing the accuracy of the normal vector of the side surface. In order to obtain a weight value that increases as the accuracy increases, for example, the reciprocal of the third eigenvalue may be used.
According to the above aspect, the direction of gravity can be determined by a weighted average using the weight values thus obtained. Therefore, since it is possible to place importance on the normal vector with high accuracy, the accuracy of the obtained gravity direction can be further improved.
The statistical method in the present invention is not necessarily limited to the above-mentioned weighted average. The weight value may be determined by the area of the side surface, the number of point groups, etc. regardless of the third eigenvalue. Further, instead of the weighted average, a simple average that does not use a weight value may be used. Further, various statistical methods such as a median value may be used instead of the average.

In the present invention,
The step (b), that is, the step of specifying a side surface point group that is a set of points representing the side surface of the feature further includes a step of specifying a horizontal component of a normal vector of each point of the side surface point group for each feature. The side surface point group may be separated into each side surface of the feature by obtaining a distribution and extracting points corresponding to the horizontal direction in which a frequency distribution equal to or higher than a predetermined value is obtained.
For example, considering a quadrangular prism-like feature, the normal vector on the side surface can be classified into two directions that are largely orthogonal. Therefore, if the points are separated according to the distribution of the normal vectors of the points, the side surfaces of the feature can be separated at least for each pair of opposing surfaces. Even in features other than the quadrangular prism shape, the side faces in various directions, and as a result, the point group can be separated into side units by the distribution of the normal vectors of the respective points.
If the point cloud is separated for each side surface in this way, it becomes easy to define the side surface including each point group, and the normal vector of the side surface can be obtained with high accuracy.

In the present invention,
In the step (b), side points for each feature are separated by separating the points where the distance between the points constituting the three-dimensional point group is a predetermined value or less as the point group constituting the same feature. It is good also as what identifies a group.
When a three-dimensional point cloud representing a plurality of features is obtained, the distance between the features is sufficiently larger than the distance between the points of the point group constituting the same feature, so if the distance between the points is used, The point cloud can be separated into feature units relatively easily. Since the side surface is a surface constituting the feature, by analyzing the side surface after being separated into feature units, the side surface can be easily defined, and the accuracy of the normal vector of the side surface can be improved.

  The present invention does not necessarily have all the features described above, and some of them may be omitted or combined as appropriate. You may comprise as a three-dimensional point cloud analysis apparatus which performs the three-dimensional point cloud analysis method mentioned above, and you may comprise as a computer program for making this analysis perform a computer. Furthermore, a computer-readable recording medium that records the computer program may be configured.

It is explanatory drawing which shows the structure of a three-dimensional model production | generation system. It is explanatory drawing which shows the aerial imaging method of an object area | region. It is a flowchart of a flight route determination process. It is a flowchart of a three-dimensional point cloud production | generation process. It is a flowchart of a three-dimensional model generation process. It is a flowchart (1) of a gravity direction analysis process. It is a flowchart (2) of a gravity direction analysis process. It is a flowchart (3) of a gravity direction analysis process. It is a flowchart (4) of a gravity direction analysis process. It is a flowchart (1) of a point group separation process. It is a flowchart (2) of a point group separation process. It is a flowchart (3) of a point group separation process. It is a flowchart (4) of a point group separation process. It is a flowchart (1) of building frame generation processing. It is a flowchart (2) of a building frame production | generation process. It is a flowchart of an alignment process. It is explanatory drawing which shows the evaluation function in a positioning process. It is explanatory drawing which shows the example of a polygon production | generation process. It is a flowchart (1) of a texture generation process. It is a flowchart (2) of a texture production | generation process.

A. System configuration:
FIG. 1 is an explanatory diagram showing a configuration of a three-dimensional model generation system. The three-dimensional model generation system is a system that takes a photograph of an area to be processed from an aircraft and generates a three-dimensional model representing a three-dimensional shape of each feature by analyzing the photograph. A device that is mounted on an aircraft and acquires image data of a local photograph is referred to as a photographing management device 100. A device that generates a three-dimensional model from image data is referred to as a three-dimensional model generation device 200. Hereinafter, each structure is demonstrated in order.

(1) Configuration of the imaging management device 100:
The imaging management device 100 is mounted on an aircraft. The aircraft may be either fixed wing or rotary wing, and may be manned / unmanned.
The photographing management apparatus 100 includes a photographing camera 101, a GPS (Global Positioning System) 102 that is a sensor for acquiring position information, and each functional block illustrated. The functional block can be configured by installing a computer program for realizing each function in the computer, or may be configured by hardware.
The camera 101 is for taking a photograph used for analysis for generating a three-dimensional model. In order to improve the analysis accuracy, a high-resolution photograph is desirable. In the present embodiment, a still image digital camera capable of shooting at a high resolution is used. If a sufficient resolution can be obtained, a video camera may be used.

The shooting sequence storage unit 112 stores flight paths and shooting points of aircraft for shooting. The shooting point is set in consideration of the angle of view of the camera 101, and is set so that 20 or more photographs are sequentially taken in the state to be processed with about 60% or more overlapping. Has been. A method for setting the flight path will be described later. The flight route or the like may be set in advance by another computer or the like, and may be stored in the imaging sequence storage unit 112 or may be set in the imaging sequence storage unit 112.
When using a manned aircraft, it is the pilot's duty to fly the aircraft according to the flight path. In such a case, the imaging sequence storage unit 112 may appropriately provide information for supporting the flight along the set flight path to the pilot.
When an unmanned aircraft is used, the imaging sequence storage unit 112 may be configured to transmit information related to the flight path to the control device that controls the flight.

The shooting control unit 110 controls the camera 101 and takes a picture of the processing target area at the shooting point set in the shooting sequence storage unit 112. Since there is an error between the actual flight path of the aircraft and the path stored in the imaging sequence storage unit 112, the imaging control unit 110 determines that the vicinity of the preset imaging point has been reached based on the error. You may make it image | photograph at the time of this.
Further, the shooting sequence is not necessarily limited to the method of specifying the shooting point, but may be set so that shooting is performed at a constant time period. In such a case, the shooting control unit 110 controls the camera 101 to perform shooting at set time intervals.
The shooting control unit 110 acquires position information of a shooting position from the GPS 102 in synchronization with shooting by the camera 101 and records the information in the image recording unit 120 in association with the shooting by any method.
The imaging control unit 110 may be configured to automatically perform imaging, and may notify an imaging person on board an imaging timing.

(2) Configuration of the three-dimensional model generation apparatus 200:
The three-dimensional model generation apparatus 200 includes the functional blocks shown in the figure. These functional blocks are configured by installing a computer program for realizing each function in a computer, or may be configured in hardware.
Hereinafter, the function of each functional block will be described.

The image data storage unit 201 reads and stores image data shot and collected by the shooting management apparatus 100 and position information of shooting positions. Transfer of image data from the imaging management apparatus 100 to the image data storage unit 201 can be performed via a wired, wireless, or recording medium.
The three-dimensional point group generation unit 202 generates a three-dimensional point group using the image data and the shooting position of the image data storage unit 201. A three-dimensional point group is a number of points to which three-dimensional coordinate values and normal vectors are assigned. The normal vector means a normal vector of a surface where each point is estimated to exist.
In this embodiment, when the image data is arranged in time series, the preceding and following image data are taken with an area of about 60% or more overlapping. That is, the building in the target area is photographed by a plurality of image data having different photographing points. Therefore, if points corresponding to each other in a plurality of image data can be specified, the three-dimensional coordinates of the points can be specified by the same principle as that of so-called triangulation. Various techniques are known as techniques for identifying corresponding points in a plurality of image data. In this embodiment, a known technique called SIFT (Scale-Invariant Feature Transform) is applied.
The three-dimensional point group data storage unit 203 stores the three-dimensional point group obtained by the three-dimensional point group generation unit 202. Since it is a point cloud obtained by analyzing a portion shown in the image data, the three-dimensional point cloud mainly represents an outer wall portion such as a building and an outer surface portion such as a roof. Also, there are many buildings other than buildings such as the ground, roads, roadside trees, vehicles, etc. in the image, and as long as the correspondence in the image is specified, a 3D point cloud can be generated. The point cloud includes many things that represent other than these buildings. Each three-dimensional point group has a three-dimensional coordinate value and a normal vector, but these coordinate value and normal vector are not defined in the earth coordinate system and are set for analysis. It is a coordinate value in a fictitious three-dimensional space.
In this way, 3D polygons with position coordinates are generated with accuracy that can be used as a map for each building from a 3D point group that is not specified in the earth coordinate system and is not divided into buildings. It is the function of each functional block shown below.

The gravity direction analysis unit 204 analyzes the gravity direction of the point group. Usually, when the geographical position is represented in the Cartesian coordinate system, the direction of gravity coincides with the Z-axis direction, but the coordinate system of the three-dimensional point group is not defined in the Earth coordinate system. The Z axis is not always in the direction of gravity. The gravitational direction analysis unit 204 functions as a first stage for associating the coordinate system of the point group with the earth coordinate system by obtaining the gravitational direction by analysis.
The point cloud separation processing unit 205 separates the point cloud into building units. The three-dimensional point cloud is obtained by obtaining the three-dimensional coordinates of the points shown in the image, and is not separated from the building ground or the like unless it is separated in units of buildings. The point group separation processing unit 205 has a function of analyzing the three-dimensional point group and separating it into point groups for each building.
The building frame generation unit 206 generates a building frame representing a two-dimensional outer shape based on the point group separated for each building.
The alignment processing unit 207 has a function of associating a coordinate system other than the gravity direction with the earth coordinate system by fitting a plurality of building frames to the two-dimensional map stored in the two-dimensional map data 210. The alignment processing unit 207 does not adapt the point group separated for each building to the two-dimensional map one by one. The whole of a plurality of buildings represented by point clouds is fitted to a two-dimensional map as a whole while maintaining the relative relationship in general. In this way, alignment can be performed with high accuracy by adapting the entire plurality of buildings.
The polygon generation unit 208 generates side and top polygons based on the point group for each building whose two-dimensional shape is determined by the above processing.
The texture generation unit 209 generates a texture to be pasted on the surface of the generated polygon based on the image data stored in the image data storage unit 201.
The three-dimensional polygon and texture generated by the above processing are stored in the three-dimensional model data 211.

B. Aerial view of the area of interest:
Next, a method for setting a flight path and a shooting point for shooting an image used for analysis will be described.
FIG. 2 is an explanatory diagram showing an aerial imaging method of the target area. In the present embodiment, a three-dimensional model is generated for a plurality of buildings included in a predetermined area around a target building. In general, the cityscape is often developed around landmarks such as stations. Therefore, by taking a picture of its surroundings and generating a 3D model, it is possible to efficiently create buildings corresponding to these landmarks. A three-dimensional model can be generated.

  As shown in FIG. 2A, in this embodiment, an image of a target region that is a generation target of a three-dimensional model is taken using an aircraft. The aircraft may be either a fixed wing or a rotary wing. The flight path draws a circular orbit around the target building. By drawing a circle, the region to be generated can be photographed uniformly from many directions, so that the portion that cannot be seen behind the other buildings can be reduced, and the generation accuracy of the three-dimensional model can be improved. The flight path is not necessarily limited to a circle as long as it is a trajectory that goes around the target building, and may be an ellipse or other curved shape.

FIG. 2B shows a method of determining a flight path when a circular orbit is used. In the three-dimensional restoration, the three-dimensional position of each feature point is obtained by specifying the correspondence between feature points photographed in common in a plurality of images. In order to obtain such a correspondence with high accuracy, it is necessary to set the flight path and shooting points so that each feature point is shot in many images. In this embodiment, the flight path and the shooting point are set so that the shot images overlap with each other with an area of about 60% or more. That is, as shown in FIG. 2B, the image IMG1 taken from a certain shooting point P1 and the image IMG2 taken at the next shooting point P2 overlap in a range of about 60% or more. In the example shown in the figure, the images IMG1 and IMG2 are in a state of being rotated around the target. However, the target may be shifted from the center.
If the angle of view of the camera used for shooting and the flight altitude are determined, the approximate radius of the flight path for shooting a predetermined range including the target can be obtained.

  It is desirable that the number of images taken during one round around the target be 20 or more. Therefore, it is desirable that the angle θ between the photographing points P1 and P2 is 18 degrees or less. Even if the number of shots is too large, the number of useless images that cannot be used for analysis increases, so a camera for shooting still images is used rather than shooting a large number of frames using a camera for shooting moving images. Therefore, it is preferable to take the required number of high-resolution images.

FIG. 3 is a flowchart of the flight path determination process. It is the process for determining the flight path demonstrated in FIG. This process may be performed in advance by a ground computer before the flight, or may be executed by the imaging management apparatus 100 mounted on the aircraft during the flight.
When the process is started, the computer first inputs a shooting target point (step S1). Also, the camera mounting angle r and field angle are input (step S2). The camera mounting angle r is shown in the figure. When shooting around the target as in this embodiment, the camera is mounted tilted toward the center of the flight path rather than being mounted with the center axis of the angle of view of the camera facing vertically downward. May be more efficient. The angle formed by the vertical axis of the aircraft and the central axis of the angle of view of the camera is the mounting angle r. The mounting angle r affects the shooting range of the camera.
The computer inputs the photographing altitude H (step S3). Then, the radius R of the flight path is calculated using the value input above (step S4). The method for calculating the radius R is shown in the figure. The radius R corresponding to the flight altitude H is calculated so that the shooting target falls within the shooting range in consideration of the camera angle of view and the mounting angle r. For example, when shooting is performed so that the shooting target is at the center of the shooting range, the radius R of the flight path is obtained by R = H × tan (r).
As shown in FIG. 2B, the shooting point may be set to have a predetermined angular interval θ as shown in FIG. 2B, or may be set to take images at a predetermined time interval. .

  The information on the flight path and the shooting point set in this way is stored in the shooting sequence storage unit 112 of the shooting management apparatus 100 and used during shooting. For example, when the aircraft is unmanned, it can be used for automatic control of the aircraft and the camera. When the aircraft is manned, it is the pilot's duty to maintain the flight path, and therefore it can be utilized by a method of providing the pilot with information indicating the flight path and shooting points as appropriate.

According to the aerial imaging method of the present embodiment described above, the following advantages can be obtained by taking a flight path that goes around the target.
In this embodiment, in order to analyze the correspondence of feature points between images, it is necessary that the images be taken in duplicate. It is possible to efficiently shoot images from various fields without waste while sufficiently overlapping the images.
In addition, there is an advantage that the recognition accuracy of the correspondence between the feature points between images is improved. That is, when the feature points are recognized between the images, the color components of the images are used. Therefore, if the lighting conditions at the time of shooting are different, the recognition accuracy of the correspondence is lowered. On the other hand, with the circular orbit as in this embodiment, it is possible to shoot the area around the target intensively in a relatively short time, so the change in lighting conditions is reduced and the recognition accuracy is reduced. Can be improved.
Furthermore, the target and the target area can be photographed from almost all directions without omission.

C. Three-dimensional point cloud generation processing:
When the flight path and shooting point are determined, shooting is performed while flying around the target. The captured image data is sequentially recorded in the image recording unit 120 of the imaging management apparatus 100 together with the position coordinates of the imaging point as described with reference to FIG. When the photographing is completed, the image data and the position coordinates are input to the three-dimensional model generation apparatus 200, and a three-dimensional point group is generated. The image data can be read into the 3D model generation apparatus 200 by various methods. In this embodiment, the image data is read via a recording medium.

FIG. 4 is a flowchart of the three-dimensional point group generation process. This is a process executed by the 3D point group generation unit 202 of the 3D model generation apparatus 200, and is a process executed by the CPU of the 3D model generation apparatus 200 in terms of hardware.
When the process is started, the three-dimensional model generation apparatus 200 reads image data and position information (step S10). Then, feature points are correlated between images by SIFT (Scale-Invariant Feature Transform) matching (step S11). The state of the feature point association is shown in the figure. As shown in the figure, feature points (four crosses in the figure) are extracted from each of image 1 and image 2, and the correspondence between the feature points is recognized. SIFT matching is an analysis technique for extracting feature points based on color components in an image and obtaining a correspondence relationship between the images. Since the analysis principle and algorithm are well known, detailed description thereof is omitted. In step S11, as long as the feature points between the images are associated with each other, not only SIFT matching but also various methods can be used.

When the correspondence between the feature points is obtained in this way, the three-dimensional coordinates and normal vectors of each feature point can be calculated (step S12). The calculation principle is shown in the figure. Each image used in the embodiment is provided with shooting point position information, so two or more shooting points 1 and 2 correspond to feature points for which a correspondence relationship is obtained between a plurality of images. Will be attached. If the position of the feature point in the image taken based on the angle of view from the shooting points 1 and 2 is analyzed, the direction in which the feature point was viewed from each of the shooting points 1 and 2 can be determined. It is possible to specify the relative positional relationship of the feature points with respect to the coordinate values of 1 and 2. The position coordinates of the feature points obtained here are not absolute position coordinates in the earth coordinate system, but are coordinate values on fictitious three-dimensional coordinates (xa, ya, za) set in the space for analysis. is there.
The normal vector is a vector in the normal direction of the estimated surface where each feature point is estimated to exist. In the embodiment, on the premise that a large number of feature points are extracted from corresponding images and the relative positional relationship between the feature points in each image is maintained, the correspondence relationship of the feature points between the images is recognized. . In this way, the estimated surface where the feature point exists is specified from the relative positional relationship between the feature points, and the normal vector can be obtained.
The calculation of the relative positional relationship between these images, the three-dimensional coordinates, and the normal vector is also well known as an SfM (Structure from Motion) technique and a technique using the SfM (Structure from Motion) technique.
A set of feature points for which three-dimensional coordinates and normal vectors are calculated is hereinafter referred to as a three-dimensional point group. The three-dimensional model generation apparatus 200 stores the obtained three-dimensional point group data (step S13), and ends the three-dimensional point group generation process.
In the present embodiment, a three-dimensional point group is acquired from captured image data, but the following processing can also be performed using a three-dimensional point group obtained by another method such as laser measurement.

D. 3D model generation processing:
FIG. 5 is a flowchart of the three-dimensional model generation process. This is a process for generating a three-dimensional model representing a three-dimensional shape for each building based on the three-dimensional point cloud data described above. This process is realized by the gravity direction analysis unit 204, the point cloud separation processing unit 205, the building frame generation unit 206, the alignment processing unit 207, the polygon generation unit 208, and the texture generation unit 209 shown in FIG. It is. In terms of hardware, this is a process executed by the CPU of the three-dimensional model generation apparatus 200.

FIG. 5 shows the overall processing flow. The details of each process will be described later.
In the three-dimensional model generation process, a gravity direction analysis process is first performed (step S20). This is processing corresponding to the function of the gravity direction analysis unit 204, and is processing for analyzing the three-dimensional point group and specifying the gravity direction.
Next, a point group separation process is performed (step S21). This is a process corresponding to the function of the point group separation processing unit 205, and is a process of separating a three-dimensional point group into building units.
Next, building frame generation processing is performed (step S22). This is processing corresponding to the function of the building frame generation unit 206, and is processing for specifying the two-dimensional shape of the building.
Next, alignment processing is performed (step S23). This is a process corresponding to the function of the alignment processing unit 207, and is a process for adapting a three-dimensional point group to a two-dimensional map.
Next, polygon generation processing is performed (step S24). This is a process corresponding to the function of the polygon generation unit 208, and is a process of generating side and top polygons based on a three-dimensional point group for each building.
Finally, texture generation processing is performed (step S25). This is a process corresponding to the function of the texture generation unit 209, and is a process of generating a texture to be pasted on the surface of a polygon using photographed image data.
Through the above processing, a three-dimensional model for each building having position accuracy that can be used as a three-dimensional map is generated.
Hereinafter, the contents of each process will be described in detail.

D1. Gravity direction analysis processing:
6 to 9 are flowcharts of the gravity direction analysis process. This corresponds to the process in step S20 of the three-dimensional model generation process (FIG. 5).
When the process is started, the 3D model generation apparatus 200 reads the 3D point cloud data (step S31).
Then, a principal component analysis of the entire three-dimensional point group is performed, and the third principal component is set as a temporary gravity direction Gt (step S32). The outline of principal component analysis is shown in the figure. The three-dimensional point group of this embodiment is obtained by photographing a peripheral region centered on the target as shown in FIG. Even if the filmed building is a high-rise building, its height is small compared to the horizontal extent of the filmed region. Therefore, the three-dimensional point group is distributed in a relatively flat region as shown in the drawing as a whole.
The principal component analysis is a method for obtaining a direction characterizing the distribution of the three-dimensional point cloud in the three-dimensional space. The first principal component, the second principal component, the third principal component are analyzed in order from the direction in which the distribution of the three-dimensional point cloud is large. Becomes the main component. For a three-dimensional point group distributed in a flat region, the first principal component and the second principal component are obtained based on the distribution in the horizontal direction, and the direction orthogonal thereto is the third principal component. Therefore, the third principal component represents the gravity direction Gt of the three-dimensional point group. However, since an error based on the distribution of the point cloud is included, the third principal component merely indicates the provisional gravity direction.
In this embodiment, since a three-dimensional point group distributed in a flat area is used, there is no need to perform complicated preprocessing such as extracting only a surface that is assumed to be in the vertical direction. It is possible to determine the direction of gravity.

Next, the three-dimensional model generation apparatus 200 excludes a point group whose angle θ with respect to the provisional gravity direction Gt is equal to or less than the threshold value THa (step S33). The three-dimensional point group of the embodiment has a normal vector as well as three-dimensional position coordinates. An angle θ between the normal vector and the gravity direction Gt is obtained in the range of 0 to 180 degrees, and a point group having this value equal to or less than the threshold value THa, that is, a point group having a normal vector in a direction relatively along the gravity direction. Exclude it. The threshold value THa may be set to a value that can exclude a point group in which the normal vector is substantially in the gravitational direction in consideration of an error in the normal vector of the point group and an error in the provisional gravity direction.
The state of processing is shown in the figure. In the figure, x and ◯ indicate a three-dimensional point group, respectively. Since the point cloud of an Example is obtained from the image which image | photographed the building, it distributes along the outer wall of a building. The point P1 is a point included in the horizontal plane S1 of the building, and the normal vector n1 is upward. The angle θ1 between the normal vector n1 and the provisional gravity direction Gt is smaller than the threshold value THa and is excluded in the process of step S33. Similarly, other point groups indicated by x existing on the surface S1 are also excluded.
The point P2 is a point included in the side surface S2 of the building, and the normal vector n2 faces a direction close to the horizontal. The angle θ2 between the normal vector n2 and the provisional gravity direction Gt is larger than the threshold value THa and is not excluded in the process of step S33. Similarly, other point groups indicated by ◯ existing on the surface S2 are not excluded.
The point P3 is a point included in the horizontal plane S3 of the building, and the normal vector n3 is downward. Such a surface S3 corresponds to the floor surface of a building and the like and does not appear in an image photographed from the outside. Therefore, such a point cloud cannot be obtained in the processing in the embodiment. However, here, for convenience of explanation, an example in which the normal vector n3 is downward is also shown. Also for the point P3, the angle θ1 between the normal vector n3 and the provisional gravity direction Gt is smaller than the threshold value THa, and is therefore excluded in the process of step S33. Similarly, other point groups indicated by x existing on the surface S3 are also excluded.
As mentioned above, in the process of step S33, the point group which comprises a horizontal surface can be excluded among the point groups which comprise a building.

Moving to FIG. 7, the three-dimensional model generation apparatus 200 performs clustering based on the distance between points for the extracted point group (step S34). That is, the point groups are separated in units of buildings by clustering point groups that are relatively close to each other.
The state of processing is shown in the figure. The diagram on the left schematically shows the clustering method. The three-dimensional model generation apparatus 200 arbitrarily selects one point P1 from the point group to be processed, and assigns a unique label (“A” in the example in the figure). Then, a point within a predetermined distance ra from this point P1 is searched, and the same label is given to them. In the illustrated example, the label “A” is assigned to the points P2 and P3. The distance ra is a distance that is a criterion for determining whether or not to perform clustering. Since the purpose of clustering is to separate the point group for each building, the distance ra can be arbitrarily set within a range smaller than the interval between buildings in the region to be processed.
When labels are attached to the points P2 and P3, the three-dimensional model generation apparatus 200 searches for points within the distance ra from these points and assigns these same labels. In the illustrated example, the label “A” is assigned to the points P4 and P5. By repeatedly executing such processing, it is possible to search for points that are within the distance ra from each other, and to give them the same label. When a point within the distance ra can no longer be searched, an unprocessed point to which no label is attached is selected, and a similar process is performed while assigning a new label (for example, “B”). Thus, clustering based on the distance between points can be performed.
The clustering results are illustrated on the right side of the figure. As shown in the figure, the point group can be separated into building units such as a point group belonging to the building 1 (× in the figure) and a point group belonging to the building 2 (◯ in the figure).

Next, the three-dimensional model generation apparatus 200 creates a horizontal distribution of normal vectors of point groups for each building (step S35). Since the point group excluding the horizontal plane of the building, that is, the point group constituting the side surface, is extracted by the process of step S33 and separated for each building by the process of step S34, by creating the normal vector distribution thereof, The normal vector of the side surface can be obtained.
The state of processing is shown in the figure. On the left side, a method for determining the direction of the normal vector is shown. The three-dimensional model generation apparatus 200 calculates a horizontal component nh1 for the normal line n1 of an arbitrary point P1 of the building to be processed. The horizontal component here means a component orthogonal to the provisional gravity direction Gt. The direction of the horizontal component nh1 is represented by an angle Ag1 from the reference axis. This angle Ag1 was determined in the range of 0 to 180 degrees as shown on the right side of the figure. The angle from the reference axis is usually expressed as positive in the counterclockwise direction and negative in the clockwise direction, but here the absolute value is obtained in the range of 0 to 180 degrees due to the nature of the process of obtaining the distribution. It is. Similarly, for the point P2, an angle Ag2 representing the direction of the horizontal component nh2 of the normal vector n2 is obtained.

When the distribution of the horizontal component is obtained, the three-dimensional model generation apparatus 200 separates the surfaces based on this distribution (step S36 in FIG. 8). If it is considered that each point cloud constitutes a side surface of a building, it is considered that if the peak is obtained based on the distribution of normal vectors, they represent the normal vector of the side surface. Therefore, the point cloud can be separated for each side by clustering the point cloud for each peak.
The state of processing is shown in the figure. As shown in the lower diagram, the distribution of normal vectors can be obtained by counting the number of corresponding points in each direction of the normal vectors and by illustrating the number. In the case of a building having a planar side surface as shown in the upper diagram, a relatively clear peak occurs in the normal vector. In the example of the figure, the maximum peak 1 corresponds to the side surface on the side with the larger area, and the second peak 2 corresponds to the side surface on the side with the smaller area.
Accordingly, by extracting the point cloud corresponding to the peak 1 and peak 2 of the horizontal component of the normal vector, the building point cloud can be separated into side units. Here, in the example of the figure, the point group is extracted in a range (a part surrounded by a frame in the figure) in which the peak 1 and the peak 2 have a predetermined width. This width can be arbitrarily set within a range in which peak 1 and peak 2 can be distinguished. However, in the embodiment, since the horizontal component is obtained in the range of 0 to 180 degrees, both the front surface / rear surface on the side having the largest area are separated as clusters, and the side surface on the smaller area side is separated. Both the front and back surfaces are separated as clusters.

Next, the three-dimensional model generation apparatus 200 performs clustering based on the distance between points for each separated point group (step S37). The clustering method is the same as that in step S34. Here, the distance that is the determination criterion for clustering may be a value that can separate the front / back surfaces of the building.
The state of processing is shown in the figure. As shown on the left side, in the process of step S36, the front surface S1 and the back surface S2 of the building are recognized as one cluster. When this is clustered by the distance between points, the front surface S1 and the back surface S2 that are separated from each other are classified as shown on the right side. In the drawing, the point group on the front surface S1 is indicated by x and the point group on the back surface S2 is indicated by ◯.
By the above processing, the point cloud is separated for each side of the building.

Moving to FIG. 9, the three-dimensional model generation apparatus 200 performs principal component analysis for each surface, and calculates a third principal component Vi and a third eigenvalue ei (step S38). The third principal component is the direction in which the third largest dispersion occurs in the point cloud distribution. The third eigenvalue is a dispersion value of the third principal component.
The state of processing is shown in the figure. Since the point cloud to be processed is a point cloud that constitutes the side of the building, when the principal component analysis is performed, the first principal component and the second principal component are obtained according to the in-plane dispersion and are orthogonal to these. The direction, that is, the normal direction of the surface is the third main component. The variance value becomes the third eigenvalue. If the point cloud is obtained with high accuracy, the dispersion value in the normal direction should be very small since the point cloud should be distributed on the side surface with high accuracy. Therefore, a large third eigenvalue indicates that the variation of the point group is large in the normal direction, indicating that the accuracy of the normal is low. By executing this process for each surface, it is possible to obtain a normal vector of the surface and to obtain an evaluation value representing the accuracy.

The three-dimensional model generation apparatus 200 calculates the gravity direction G using the third principal component obtained in this way (step S39).
As shown in the figure, the gravitational direction G should be obtained by obtaining these outer products Vi × Vj using normal vectors Vi and Vj of arbitrary two side faces. However, since the normal vectors Vi and Vj each contain an error, the gravitational direction G thus obtained also contains an error. Therefore, in the embodiment, an arbitrary combination that is not parallel is selected from the obtained normal vectors, an individual gravity direction is obtained by an outer product, and a final gravity direction G is calculated by a weighted average thereof. did.
Here, the weighted average is the reciprocal of the sum (ei + ej) of the third eigenvalue representing the accuracy of the normal vector. The larger the third eigenvalue, the lower the accuracy of the normal vector. Therefore, by multiplying the inverse of the third eigenvalue as a weight value, the higher the normal vector accuracy, the greater the weight value. . Based on the above concept, in this embodiment, the direction of gravity is obtained by the average of (Vi × Vj) / (ei + ej).
The calculation of the gravity direction G is not limited to the method described above. For example, the weight value only needs to reflect the accuracy of the normal vector, and may be a product of ei and ej in addition to ei + ej. Alternatively, not all normal vectors may be used, but only normal vectors that can be considered to have a high accuracy that the third eigenvalue is equal to or lower than a predetermined threshold value may be used.

  Through the above processing, the direction of gravity can be determined based on the three-dimensional point group. According to the present embodiment, principal component analysis is applied to the determination of the provisional gravity direction (step S32 in FIG. 6), the determination of the normal vector of the side surface (step S38 in FIG. 9), and the like. By performing statistical processing in this way, it is possible to suppress the influence of errors included in the position coordinates and normal vectors of each point, and to determine the direction of gravity with high accuracy.

D2. Point cloud separation processing:
10 to 13 are flowcharts of the point group separation process. This corresponds to the process in step S21 of the three-dimensional model generation process (FIG. 5).
In the gravity direction analysis process executed earlier, the process of separating the point group into the building unit and further the side surface unit of the building was performed using clustering based on the distance between the points (step S34 in FIG. 7 and FIG. 8). Step S37). However, there are buildings such as a so-called twin tower in which only a high floor is divided into a plurality of buildings, and only an intermediate hierarchy is divided into a plurality of buildings. In clustering in the gravity direction analysis processing, the point cloud cannot be separated into building units for such a specially shaped building. In the point group separation process, the process of separating the point group into building units is executed in consideration of the existence of such buildings. Although the point group separation process can target only a specially shaped building, it is not easy to extract a building having such a shape from a three-dimensional point group. However, processing is performed for all buildings regardless of whether the shape is special or not.

  When the process starts, the three-dimensional model generation apparatus 200 reads point cloud data to be processed (step S50). The point group used as a process target was illustrated in the figure. Since the density of point clouds is sufficiently high, depending on the display scale, even if only point clouds like this can be seen as if the shape of the building has already been completed, the point clouds are not associated with each other at all, It is just a collection of points that are not separated for each building.

The three-dimensional model generation apparatus 200 separates the read point group for each hierarchy (step 51). The height H of the hierarchy can be arbitrarily determined. Even if the building has a special shape, such as a high-rise building that is divided into multiple branches, it is normal for the building to be branched at the boundary between the floor and the floor. H is preferably a value corresponding to the height of each floor of a general building.
In the figure, the state of separation for each hierarchy is shown. X in the figure represents a three-dimensional point group. Separation into hierarchies is performed by setting a boundary plane for each predetermined height H in a three-dimensional space in which a three-dimensional point group exists. In the example shown in the figure, the boundary plane of the hierarchy is positioned on the roof surface of the left building, but the boundary surface may be set regardless of the building shape.

Next, the three-dimensional model generation apparatus 200 performs clustering based on the distance between points in the hierarchy (step S52). The clustering method is the same as that described in the gravity direction analysis process (see step S34 in FIG. 7).
The state of processing is shown in the figure. Consider a case in which a point group indicated by x exists in the hierarchy to be processed. When clustering is performed according to the distance, the point group is separated into the left and right clusters CL1 and CL2. The reason why the point cloud corresponding to the rooftop is included only in the cluster CL1 is that the building on the cluster CL2 side is higher than the layer to be processed, and this layer includes only the side point cloud. Because.

Moving to FIG. 11, the three-dimensional model generation device 200 calculates the distribution of upward normals for each separated cluster (step S53). For example, as shown on the left side of the figure, the upward normal is the absolute value | n · G | of the inner product of the point normal vector n and the gravity direction G (hereinafter referred to as “evaluation value” in the description of this processing). Can also be evaluated). If this evaluation value is equal to or greater than a predetermined value, it can be determined that the normal is close to the gravity direction G and is an upward normal.
The distribution of upward normals is shown on the right side of the figure. As a result of the cluster CL2 in which the point group exists only on the side surface, as indicated by the solid line, the ratio of the points where the evaluation value | n · G | On the other hand, as shown by the broken line, the result of the cluster CL1 including the point group on the roof surface is the ratio of the points where the evaluation value | n · G | Two points of the point ratio become high.

The three-dimensional model generation apparatus 200 assigns an attribute to each cluster according to the distribution thus obtained (step S54). That is, when the ratio of the upward normal line exceeds a predetermined threshold value THV, it is defined as the upper surface, and otherwise it is defined as the side surface. The upper surface does not necessarily mean that the point group is composed only of the upper surface, but means that the point group includes the upper surface. According to the example shown in step S53, the cluster CL1 is defined as the upper surface, and the cluster CL2 is defined as the side surface. The threshold value THV used in the definition is a value used as a criterion for determining whether or not the upper surface exists, and is arbitrarily set based on an error included in the normal vector of the point cloud or an area of the upper surface of a normal building. Can do.
The attribute assignment result is shown in the drawing of step S54. Here, an independent rectangular parallelepiped building and a building that branches into two only on the upper floors are illustrated. What is drawn as a stack of boxes is a schematic representation of the result of separation into layers. In the drawing, the upper surface is hatched and the side surface is white. As shown in the figure, in an independent building, the level R1 including the roof surface is determined as the upper surface. In a branched building, the top surfaces R2 and R3 of the branching portion and the middle layer R4 corresponding to the root of the branch are determined as the top surface.
Furthermore, in this embodiment, the ground surface is also determined as the upper surface. This is because in the three-dimensional point group, the building and the ground surface are not separated, and the ground surface corresponds to a surface having a high ratio of upward normals.

Moving to FIG. 12, the three-dimensional model generation apparatus 200 calculates the distance between clusters in adjacent layers (step S55). The cluster attributes (upper surface and side surface) previously assigned need not be considered in this processing.
The state of processing is shown in the figure. Here, calculation of the distance between the cluster CLU belonging to the upper hierarchy and the clusters CLL1 and CLL2 belonging to the lower hierarchy will be described. For humans, it can be easily identified that the cluster CLU and the cluster CLL2 are clusters located above and below the same building. However, in the process of analyzing the three-dimensional point cloud, the computer makes such a determination. The algorithm is necessary. Since the calculation of the distance between the clusters in the adjacent hierarchy is a process performed to specify the clusters arranged above and below the same building, the cluster CLU belonging to the upper hierarchy targets all the clusters CLL1 and CLL2 belonging to the lower hierarchy. In addition, the distance between the clusters is calculated.
The three-dimensional model generation apparatus 200 first projects the point group of each cluster CLU, CLL1, CLL2 to be processed on a horizontal plane. Then, in the projection plane, the distance between the points belonging to the cluster CLU and the points belonging to the cluster CLL1 is calculated, and the minimum value of the distance is set as the distance between both clusters. That is, since there are many points belonging to the cluster CLU (◯ in the figure) and points (x in the figure) belonging to the cluster CLL1, one point is arbitrarily selected from each, and the distance between the points is calculated. The distance between the cluster CLU and the cluster CLL1 can be obtained by calculating the distance for all combinations of points and obtaining the minimum value among them. In the illustrated example, the distance D11 between the point P1 in the cluster CLU and the point P3 in the cluster CLL1 is the distance between both clusters.
In the same way, the three-dimensional model generation device 200 calculates the distance between the cluster CLU and the cluster CLL2. In the example of the figure, the distance D12 between the point P1 in the cluster CLU and the point P2 in the cluster CLL1 is the distance between both clusters.
Since the clusters CLL1 and CLL2 are clusters belonging to the same hierarchy, it is not necessary to calculate the distance between the clusters.

Next, the three-dimensional model generation apparatus 200 determines that the clusters satisfying the calculated inter-cluster distance <Thd are overlapped in the same building, and associates them (step S56).
The state of processing is shown in the figure. When viewed from the cluster CLU, it is assumed that the intercluster distance D11 between the cluster CLL1 is larger than the threshold Thd and the intercluster distance D12 between the clusters CLL2 is smaller than the threshold Thd. As a result, with respect to the cluster CLU, the cluster CLL2 is associated with the same building in the vertical relationship, and the cluster CLL1 is not associated.
The threshold value Thd is a reference value for determining the vertical relationship in this way. The value can be arbitrarily set in a range smaller than the interval between the branch portions of the building.
In the embodiment, the ground surface exists in the lowest layer, and the intercluster distance between the building and the ground surface is small. Therefore, all the buildings are associated with the ground surface.

Moving to FIG. 13, the three-dimensional model generation apparatus 200 associates all clusters and generates a tree structure representing the entire association (step S57).
An example of a tree structure is shown in the figure. Taking the building shown on the left as an example, the tree structure is shown on the right. About the building LV3 constructed independently, each cluster is associated as a single line without branching or joining, and reaches the ground surface LV5.
As for the buildings that are branched, after being linked in the state of branching from the branch portions LV1 and LV2 on the higher floor to the lower floor, they merge at the middle level LV4, and then one on the lower floor Each cluster is associated as a streak and reaches the ground surface LV5. For other buildings, a tree structure is generated by the same method.
In the figure, the tree structure is illustrated, but the generation of the tree structure in step S57 does not necessarily mean drawing such a figure, but means specifying the association between all clusters. Yes.

  Next, the three-dimensional model generation apparatus 200 refers to the tree structure and determines that the lowest layer cluster is the ground surface (step S58). When it is known that the ground surface is included in the three-dimensional point group, a method of determining the lowest layer as the ground surface may be taken as preprocessing. If the method of determining the lowest layer in associating the upper and lower relationships as in the present embodiment, it is possible to specify the ground surface even when the presence of the ground surface cannot be confirmed.

Next, the three-dimensional model generation apparatus 200 separates the point group in units of clusters adjacent to the ground surface (step S59). Considering that all buildings are ultimately associated with the ground surface, the clusters adjacent to the ground surface, that is, the clusters in the hierarchy immediately above the ground surface, are considered to represent the basic portion of the building. Therefore, if the point cloud is separated for each layer of the foundation portion, the point cloud can be separated for each building.
The state of processing is shown in the figure. Except for the ground surface of the lowest layer (broken line in the figure), it is determined that the levels CLR1 and CLR2 directly above correspond to the basic part of the building. Therefore, it is determined that the list of clusters associated with the hierarchy CLR1 represents the building 1. In addition, a series of clusters associated with the hierarchy CLR2 is determined to represent the building 2 including a part that branches in the middle.
By using the cluster association and the tree structure in this way, it is possible to separate a three-dimensional point group for each building, including specially shaped buildings.

D3. Building frame generation processing:
14 and 15 are flowcharts of the building frame generation process. This corresponds to the process in step S22 of the three-dimensional model generation process (FIG. 5). This is processing for specifying a two-dimensional shape of a building based on a three-dimensional point group separated for each building.
The three-dimensional model generation device 200 reads the three-dimensional point cloud data of the building to be processed (Step S70).
Then, based on the normal vector of the point group, the top point group and the side point group are separated (step S71).
The state of processing is shown in the figure. The point group on the upper surface has a normal vector facing upward as indicated by a point P1, and the point group on the side surface has a normal vector substantially facing a horizontal direction as indicated by a point P2. A point cloud having an upward normal vector can be identified by a method similar to the point cloud separation process (see step S53 in FIG. 11).
In the example in the figure, the upper surface 1 and the upper surface 2 of two different heights (the hatched portion in the figure) are separated from the other side surfaces of the building. The building frame generation processing is normally performed based on the idea that the two-dimensional shape of the building will be a shape that encloses the upper surface. Therefore, the upper surfaces separated in step S70 may be surfaces having different heights such as the upper surface 1 and the upper surface 2 shown in the drawing.

The three-dimensional model generation apparatus 200 arranges the grid along the separated side surfaces (step S72).
The state of processing is shown in the figure. A large number of point clouds exist on the separated side surfaces. When these point groups are projected onto a horizontal plane, a plane shape of the building, that is, a point group distributed along the building frame is obtained. Although it is possible to obtain line segments passing through these many point groups by analysis, in this embodiment, as a simpler method, points to be considered when determining the building frame by using a grid. The group was to be reduced.
An example in which a grid is arranged on the right side of the figure is shown. The grid can be arranged in an arbitrary state with respect to the building. However, in order to accurately represent the shape of the building frame, the grid GL2 is more preferable than the case where the grid is arranged obliquely like the grid GL1. It is preferable to arrange | position along the longest side of a building frame like.
As a method of performing such arrangement, for example, a method of determining the arrangement direction of the grid GL2 by analyzing the point cloud of the building frame to obtain the direction of the longest side can be used.
Further, as another method, the following two-stage method may be used. First, after temporarily arranging the grid, a grid to which the point group belongs is extracted, and the longest length in which the grid is arranged in the vertical or horizontal direction of the grid is obtained as an evaluation value. Second, the evaluation value is obtained in the same manner while rotating the grid arrangement direction, and the direction in which the evaluation value is maximized is obtained. In this way, the grid can be arranged along the building frame.
The grid size of the grid can be set arbitrarily. If the lattice size is small, the building shape can be expressed with high accuracy, while the processing load increases. If the lattice size is large, the accuracy of representing the building shape becomes coarse, but the processing load is reduced. The lattice size may be determined in consideration of these.

Moving to FIG. 15, the three-dimensional model generation device 200 projects a three-dimensional point group on the upper surface onto the arranged grid (step S <b> 73).
The point cloud is projected in the figure. The points are represented by x. A grid in which one or a plurality of point groups exist in each grid, or a grid in which no points exist at all appears.
Instead of the top surface, a point cloud on the side surface may be projected. However, in a building that branches off from the middle (see step S54 in FIG. 11) or a building whose upper surface projects beyond the side surface, It is preferable to use the upper surface because there may be cases where it cannot be specified.

Next, the three-dimensional model generation apparatus 200 determines a building frame by extracting a grid in which point clouds are present (step S74).
The state of processing is shown in the figure. Based on the projection result of the point group shown in step S73, a grid in which one or more points exist is extracted. In the figure in step S74, such a lattice is shown with hatching. The three-dimensional model generation apparatus 200 specifies a building frame by connecting the representative points of the hatched grids. In this example, the center of gravity of the lattice (the point indicated by ● in the figure) was used as the representative point.
By this processing, the number of points to be processed is reduced to the number of grid points at the position corresponding to the building frame.

The three-dimensional model generation apparatus 200 further performs a process of thinning out the vertices (step S75).
In this embodiment, as shown on the left side in the figure, a method of deleting a vertex located at the center of a portion determined to be arranged substantially linearly is adopted. For example, when the vertices a, b, and c exist, the outer angle α of the line segments ab and bc is obtained. If α <threshold, it is determined that these three points are arranged almost linearly, and the vertex b is deleted. The threshold value is a value serving as a criterion for determining whether or not to delete the vertex b, and can be arbitrarily set from the viewpoint of whether or not a significant shape change is given to the building frame by deleting the vertex b.
The result of deleting a vertex is illustrated on the right side of the figure. With this process, the shape shown in step S74 can be deleted at the four points of vertices P1 to P4 as shown in the drawing of step S75.
Through the above processing, the three-dimensional model generation apparatus 200 can determine a building frame.

D4. Alignment process:
FIG. 16 is a flowchart of the alignment process. This corresponds to the process in step S23 of the three-dimensional model generation process (FIG. 5).
By the processing described above, the three-dimensional point group is divided into building units, and the building frame is also required. However, the building frame obtained in this way is based on the point cloud analysis and includes the point cloud position error and analysis error. Therefore, the building frame is not necessarily accurate enough to be used as a three-dimensional map model. Not. In the alignment process, the accuracy of the shape and position of the building frame is improved by referring to the two-dimensional map data.
Here, the concept of processing in this embodiment will be described first. The alignment processing in the present embodiment does not perform fine correction of the shape and position in comparison with the two-dimensional map data for each building. In this embodiment, three-dimensional point clouds are obtained for the area around the target building, and these three-dimensional point clouds are relative to each other with the same accuracy as reproducing the top surface and side surface of each building. It should also represent a typical positional relationship. Under this concept, the present embodiment finds a shape and position conversion method that can most accurately fit the whole of the plurality of building frames obtained by the above processing to the two-dimensional map.
In the method of matching the position and shape for each building, information held by the three-dimensional point group is ignored regarding the relative positional relationship of the buildings. On the other hand, according to the method of the present embodiment, the information of the three-dimensional point group can be fully utilized for the shape of each building and the relative positional relationship of the buildings, so that the alignment can be performed with high accuracy. It is possible to do this. The processing method will be described below.

When the process is started, the 3D model generation apparatus 200 reads 3D point cloud data and 2D map data (step S80).
Then, the end point position of the side surface is calculated based on the three-dimensional point group (step S81).
The state of processing is shown in the figure. A normal vector n representing the side surface is obtained based on the side point cloud. The normal vector n can be obtained by the same method as the gravity direction analysis process (see step S38 in FIG. 9).
Further, the direction orthogonal to these, that is, the width direction of the side surface is obtained by the outer product g × n of the gravity direction g and the normal vector n. In this way, the points W1 and W2 at which the point group on the side surface is located farthest in the width direction are obtained and set as the end point positions on the side surface. Here, since the three-dimensional point group is scattered, the points W1 and W2 are not always present on the line segment in the width direction. Therefore, it is possible to take a method of obtaining the points W1 and W2 after projecting the point group constituting the side surface onto the line segment in the width direction.

Next, the three-dimensional model generation apparatus 200 performs alignment so that the end point position of the side surface is aligned with the two-dimensional map (step S82).
The state of processing is shown in the figure. A broken line on the left side represents the shape of the building frame obtained by the distribution of the three-dimensional point group, and a solid line and points on the right side represent the building frame and its apex on the two-dimensional map.
In this example, an example in which end points W1, W2, W3, and W4 of side surfaces obtained from two buildings in the three-dimensional point group are matched with a two-dimensional map is shown. First, the three-dimensional model generation apparatus 200 arbitrarily associates the end points W1 to W4 with vertices on the two-dimensional map. If a human sees the 3D point cloud and the 2D map, the correspondence between the end points W1 to W4 and the vertices on the 2D map can be easily specified. Since there is no information for associating with each other, information arbitrarily selected from both sides is associated with each other.
For example, it is assumed that the end points W1, W2, W3, and W4 are associated with the vertices f1, f2, f5, and f6, respectively. Next, the three-dimensional model generation apparatus 200 obtains a conversion formula for converting the position of the three-dimensional point group that matches this association. In this embodiment, a transformation matrix for homography transformation is obtained. Then, an evaluation value indicating how much the shape and position of the three-dimensional point group is deformed by such conversion is obtained. A method for calculating the evaluation value will be described later.
The three-dimensional model generation apparatus 200 repeats the above-described conversion matrix calculation and evaluation value calculation while changing the correspondence between the end points and the vertices. And the correspondence which can make a three-dimensional point group adapt to a two-dimensional map by the smallest deformation is specified.
In the example shown in the figure, when the end points W1, W2, W3, and W4 are associated with the vertices f1, f2, f5, and f6, in the transformation that realizes this, the shape and position of the three-dimensional point group are greatly deformed. On the other hand, when the end points W1, W2, W3, and W4 are associated with the vertices f1, f2, f3, and f4, in the transformation that realizes this, the deformation of the shape and position of the three-dimensional point group is small. As a result, the end points W1, W2, W3, and W4 can be specified as corresponding to the vertices f1, f2, f3, and f4.
Based on this way of thinking, a method for specifying an optimum correspondence is called an annealing method or a simulated annealing method. The correspondence relationship may be specified using another method.

FIG. 17 is an explanatory diagram showing evaluation values in the alignment process.
As described above, in the alignment process, an evaluation value for evaluating the magnitude of deformation when a three-dimensional point group is aligned with a two-dimensional map is used.
FIG. 17A shows a method for calculating an evaluation value using square ABCD. In step S82 in FIG. 16, when a conversion method from the end points of the three-dimensional point group to the vertices of the two-dimensional map is obtained, squares A1, B1, C1, and D1 are obtained by converting the square ABCD by this conversion method. . The smaller the deformation due to the conversion method, the deformed squares A1, B1, C1, and D1 should have a shape closer to the original square ABCD. Therefore, the change in shape from the square can be used as an evaluation value for this conversion method. As such an evaluation value, for example, the following evaluation value Ea1 can be considered.
Evaluation value Ea1 = | A1 / (π / 2) -1 | + | B1 / (π / 2) -1 | + | C1 / (π / 2) -1 | + | D1 / (π / 2) -1 ｜;
Here, A1, B1, C1, and D1 represent the magnitude (rad) of each apex angle.
The evaluation value Ea1 is an evaluation value paying attention to a deviation from 90 degrees (π / 2 (rad)) of the apex angles of the quadrangles A1, B1, C1, and D1 after deformation. In this example, the evaluation values are obtained using all of the apex angles A1, B1, C1, and D1, but may be represented by any apex angle.

Moreover, you may use the following evaluation value Ea2 as another evaluation value.
Evaluation value Ea2 = | 1- (A1B1) / (C1D1) |;
Here, A1B1 represents the length of the side A1B1, and C1D1 represents the length of the side C1D1.
The evaluation value Ea2 is an evaluation value paying attention to a change in the length of opposite sides of the deformed squares A1, B1, C1, and D1. In this example, the side A1B1 and the side C1D1 are used, but other opposite sides, that is, the side A1D1 and the side B1C1 may be used together.

FIG. 17B shows a method for calculating an evaluation value based on the relative positional relationship between the end points of the three-dimensional point group and the vertices of the corresponding two-dimensional map. On the left side of the figure, end points W1 to W5 in the three-dimensional point group are shown. The center of gravity of these end points W1 to W5 is obtained as GW, and the relative positional relationship of each end point can be expressed by the angle formed by the line segment from the end point toward the center of gravity. For example, the positional relationship between the end points W1 and W2 is represented by an angle θW12 formed by the line segments W1 and GW and the line segments W2 and GW.
On the right side, vertices f1 to f5 associated with the end points W1 to W5 in the two-dimensional map are shown. The center of gravity of these vertices f1 to f5 is obtained as Gf, and the relative positional relationship between the vertices can be defined similarly to the end points. For example, the positional relationship between the vertices f1 and f2 is represented by an angle θf12 formed by the line segments f1 and Gf and the line segments f2 and Gf.
At this time, the following value can be considered as the evaluation value Eb1 representing the degree of deformation.
Evaluation value Eb1 = dispersion or standard deviation of (θfij−θWij);
The evaluation value Eb1 is an evaluation value paying attention to the magnitude of deviation in the relative positional relationship between the end points and the vertices.
As another evaluation value, the next evaluation value Eb2 may be used.
Evaluation value Eb2 = dispersion or standard deviation of | (fiGf) / (WiGW) |
Here, fiGf represents the length of the line segments fi and Gf, and WiGW represents the length of the line segments Wi and GW.
The evaluation value Eb2 is an evaluation value paying attention to a change in relative length between the end point and the vertex and the respective center of gravity positions.
Any one of the above-described evaluation values Ea1, Ea2, Eb1, and Eb2 may be selected and used. Moreover, it is good also as what calculates | requires a new evaluation value with the calculation formula using these.

  With the above processing, the three-dimensional point group can be matched with the two-dimensional map, and the correspondence between the point group representing the building and the feature on the map can be specified. In addition, according to the method of the present embodiment, the information of the three-dimensional point group can be fully utilized with respect to the shape of each building and the relative positional relationship between the buildings, so that the alignment is performed with high accuracy. Is possible.

D5. Polygon generation processing:
FIG. 18 is an explanatory diagram showing an example of polygon generation processing. This corresponds to the process in step S24 of the three-dimensional model generation process (FIG. 5).
Through the processing described above, the three-dimensional point group is separated for each building, and a building frame is also required. In addition, since the correspondence between the three-dimensional point group and the two-dimensional map is also required, the accuracy of the shape of the building frame is further improved by the homography conversion used in the correspondence.
The three-dimensional model generation apparatus 200 generates a polygon for each building from the three-dimensional point group based on the information thus obtained.
Specifically, the upper surface is defined on the basis of the building frame, and then this is translated downward to set a polygon on the side surface of the building for each side. Similarly, for a specially shaped building (see step S57 in FIG. 13), the side surfaces of the branched portions are generated by translating the branched upper surfaces downward, and the lower hierarchical upper surface is translated downward. By doing so, a low-level side polygon is generated. In addition to using the shape of the upper surface in this way, the polygon may be defined based on the point group on the side surface separated by the building frame generation process (see step S71 in FIG. 14).
FIG. 18A shows a state of a three-dimensional point group before generation, and FIG. 18B shows a state of a generated polygon. It can be seen that the three-dimensional point cloud is accurately reproduced by the polygon.

D6. Texture generation processing:
19 and 20 are flowcharts of the texture generation process. This corresponds to the process in step S25 of the three-dimensional model generation process (FIG. 5).
When the processing is started, the three-dimensional model generation apparatus 200 inputs polygon data and image data generated by the polygon generation processing, and shooting conditions (step S90). The shooting conditions include a shooting position, a camera direction at the time of shooting, a field angle, and the like.
Next, the three-dimensional model generation apparatus 200 extracts image data in which polygons are drawn in the image (step S91). This is a process for selecting an image suitable for cutting out a texture for a building. As a method for selecting such an image, it is possible to determine whether or not the target building is shown by image analysis of the photographed image, but a great processing load is applied. Therefore, in this embodiment, since the building polygon has already been obtained, this polygon is perspective-projected under the projection conditions corresponding to the photographing position and photographing condition read in step S90, and based on the obtained projection image. It is judged whether the target building is reflected.
The state of processing was illustrated in the figure. In the image 1 on the left side, when a building is drawn under a projection condition corresponding to the shooting position and shooting condition, a portion that protrudes outside the image occurs. In the image 2 on the right side, the entire building is drawn in the image. In this embodiment, based on the result of drawing the polygon in this way, the image 2 in which the entire building is drawn is extracted for cutting out the texture.

Next, the three-dimensional model generation apparatus 200 excludes image data in which the polygon to be processed is the back from the extracted images (step S92). This processing is also performed using building polygons rather than image analysis.
The state of processing is shown in the figure. In the left image, the polygon to be processed is drawn as the back side. On the other hand, the image on the right side is drawn in a visible range. Accordingly, the image data on the left side is determined to be inappropriate for cutting out the texture, and is excluded. For example, when the angle formed between the normal vector of the polygon to be processed and the vector representing the line-of-sight direction is 90 degrees or less, it can be determined that the target polygon is the back surface.

  The three-dimensional model generation apparatus 200 further excludes image data in which the processing target polygon is blocked by another polygon (step S93). For example, it is possible to take a method of determining that the side of the polygon to be processed is shielded when the side of the polygon intersects the side of another polygon.

As a candidate image for cutting out the texture by the above processing, there are those in which the building is reflected in the image, the polygon to be processed is not the back side, and the image is not shielded by other polygons. Should have been selected. That is, an image in which the entire polygon to be processed is captured without being hidden or cut is a candidate image.
The three-dimensional model generation device 200 selects the candidate image data having the largest processing target polygon area (step S94).
The state of processing is shown in the figure. As shown in the figure, when there are three candidate images, the central image obtained by photographing the processing target polygon (the hatched portion in the figure) from the front has the maximum area, so this is selected. This process can be performed by calculating the area of the polygon to be processed. In addition, since an image obtained by photographing a polygon from the front usually has the largest area, an image that minimizes the angle formed by the normal vector of the polygon and the line-of-sight direction may be selected as a candidate.
In selecting a candidate, not only the area of the polygon to be processed but also the image quality of the image data may be considered.

When the three-dimensional model generation apparatus 200 determines a candidate image for cutting out a texture in this manner, the region corresponding to the processing target polygon is cut out from the candidate image and stored as a texture (step S95).
According to the texture generation process of the present embodiment, it is possible to generate a texture with excellent image quality by using an image obtained by photographing a polygon to be processed from near the front.
Further, when selecting such an image, simulation, that is, image data obtained by performing perspective projection at a photographing position and photographing condition corresponding to each image data using a building polygon obtained from a three-dimensional point cloud. By using, it is possible to perform processing accurately with a light processing load as compared with the case of using image analysis.

The embodiment of the present invention has been described above. The present invention does not necessarily have all the functions of the above-described embodiments, and only a part may be realized. Moreover, you may provide an additional function in the content mentioned above.
Needless to say, the present invention is not limited to the above-described embodiments, and various configurations can be adopted without departing from the spirit of the present invention. For example, a part configured in hardware in the embodiment can be configured in software and vice versa.

  The present invention can be used to generate a three-dimensional model from a three-dimensional point cloud.

DESCRIPTION OF SYMBOLS 100 ... Shooting management apparatus 101 ... Camera 102 ... GPS
DESCRIPTION OF SYMBOLS 110 ... Shooting control part 112 ... Shooting sequence memory | storage part 120 ... Image recording part 200 ... Three-dimensional model production | generation apparatus 201 ... Image data memory | storage part 202 ... Three-dimensional point cloud data generation part 203 ... Three-dimensional point cloud data memory part 204 ... Gravity direction Analysis unit 205 ... Point group separation processing unit 206 ... Building frame generation unit 207 ... Alignment processing unit 208 ... Polygon generation unit 209 ... Texture generation unit 210 ... 2D map data 211 ... 3D model data

Claims (8)

A three-dimensional point cloud analysis method for analyzing a three-dimensional point cloud composed of a plurality of points representing the outer shape of a plurality of features by a computer,
3D point group data constituting the three-dimensional point group with respect to a plurality of points representing the outer shape of the feature, the position coordinates and the point in the three-dimensional coordinate space that is configured for analysis for each point there Contains the normal vector of the inferred surface ,
In the three-dimensional point cloud analysis method, the computer executes the following steps:
(A) reading the three-dimensional point cloud data;
(B) identifying a side point group that is a set of points representing the side surface of the feature from the three-dimensional point group data;
(C) obtaining a normal vector of each side of the feature based on the side point group;
(D) Obtaining an outer product of two normal vectors selected from the normal vectors for selection of a plurality of sets, and determining a direction of gravity in the three-dimensional coordinate space based on statistical processing of the obtained plural outer products. A three-dimensional point cloud analysis method. The three-dimensional point cloud analysis method according to claim 1,
The three-dimensional point group is compared with the distribution of the height of the feature existing in the target range, and the horizontal distribution of the position of the feature in the target range becomes the first principal component and the second principal component. Obtained for a sufficiently large area ,
In the three-dimensional point cloud analysis method, the computer executes the following steps:
(E) setting a direction of a third principal component obtained by principal component analysis of the three-dimensional point group as a temporary gravity direction;
The step (b) specifies the side surface point group by excluding a point group in which a normal vector of each point is along the temporary gravity direction. A three-dimensional point cloud analysis method according to claim 1 or 2,
The step (c) separates the side surface point group into each side surface of each feature, and calculates a third principal component obtained by performing principal component analysis for each side surface as a normal vector of each side surface of the feature. Yes 3D point cloud analysis method. A three-dimensional point cloud analysis method according to claim 3,
The step (d) uses the third eigenvalue obtained for the normal vector in the principal component analysis to calculate a weight value representing the accuracy of the outer product of the normal vectors in each set, and the statistical As a process, a three-dimensional point cloud analysis method for obtaining a weighted average of the outer products using the weight values. A three-dimensional point cloud analysis method according to any one of claims 1 to 4,
In the step (b), the distribution of the horizontal component of the normal vector of each point of the side surface point group is obtained for each feature, and the point corresponding to the horizontal direction in which a frequency distribution greater than or equal to a predetermined value is obtained A three-dimensional point cloud analysis method that separates the side surface point group into each side surface of the feature by extracting. A three-dimensional point cloud analysis method according to any one of claims 1 to 5,
In the step (b), side points for each feature are separated by separating the points where the distance between the points constituting the three-dimensional point group is a predetermined value or less as the point group constituting the same feature. 3D point cloud analysis method to identify groups. A three-dimensional point cloud analysis device for analyzing a three-dimensional point cloud consisting of a plurality of points representing the outer shape of a plurality of features,
As the three-dimensional point cloud data constituting the three-dimensional point cloud, with respect to a plurality of points representing the outer shape of the feature, position coordinates in the three-dimensional coordinate space set for analysis for each point and the points exist. A three-dimensional point cloud data storage unit for storing a normal vector of the estimated surface ;
A gravity direction analysis unit for determining a gravity direction in the three-dimensional coordinate space;
The gravity direction analysis unit
From the three-dimensional point cloud data, specify a side point cloud that is a set of points representing the side surface of the feature,
Based on the side point group, obtain a normal vector of each side of the feature;
A cross product of two normal vectors selected from the normal vectors is obtained for a plurality of sets of selections, and a direction of gravity in the three-dimensional coordinate space is determined based on statistical processing of the obtained cross products. Point cloud analysis device. A computer program for analyzing, by a computer, a three-dimensional point group consisting of a plurality of points representing the outline of a plurality of features,
3D point group data constituting the three-dimensional point group with respect to a plurality of points representing the outer shape of the feature, the position coordinates and the point in the three-dimensional coordinate space that is configured for analysis for each point there Contains the normal vector of the inferred surface ,
The computer program is
(A) a function of reading the three-dimensional point cloud data;
(B) a function for specifying a side surface point group that is a set of points representing the side surface of the feature from the three-dimensional point group data;
(C) a function for obtaining a normal vector of each side of the feature based on the side point group;
(D) Obtaining an outer product of two normal vectors selected from the normal vectors for selection of a plurality of sets, and determining a direction of gravity in the three-dimensional coordinate space based on statistical processing of the obtained plural outer products. A computer program that causes a computer to realize the functions to be performed.