RU2638638C1 - Method and system of automatic constructing three-dimensional models of cities - Google Patents

Abstract

FIELD: information technology.

SUBSTANCE: method of automatic constructing three-dimensional models of cities based on the processing of data from various sources, in particular laser scanning and photographic data, the formation of a single point cloud, the cleaning of a given point cloud, the sequential identification of urban objects, the construction of three-dimensional models of identified objects and their subsequent integration for constructing a single three-dimensional model of the city.

EFFECT: increasing the accuracy of constructing a three-dimensional model of the city by providing the construction of a city model based on separately recognized urban objects derived from previously cleaned laser scanning data.

13 cl, 9 dwg

Description

FIELD OF TECHNOLOGY

The claimed solution relates to the field of information processing methods, in particular to a method and system for constructing three-dimensional city models based on laser scanning information and photographic images.

BACKGROUND

To date, various solutions are known aimed at recognizing various objects from photographic information, as well as data obtained during laser scanning.

A known technology for the recognition and construction of three-dimensional models of building facades based on information obtained from clouds of laser scanning points with subsequent processing of an intermediate model and superimposing a photographic image on a three-dimensional frame (3D All The Way: Semantic Segmentation of Urban Scenes From Start to End in 3D // Andelo Martinovic et al.). This method is also used for the semantic separation of three-dimensional urban models, to determine similar objects based on a trained algorithm that allows you to recognize and mark similar objects, in particular building facades.

The known method is limited by the type of recognizable objects, and also does not have sufficient accuracy of determination, while using the known method it is impossible to build a 3D model of the city on the basis of separately identified city objects.

SUMMARY OF THE INVENTION

The technical problem that can be solved using the claimed solution is the creation of a new way to build a model of the city with increased accuracy of object recognition.

The technical result is to increase the accuracy of building a three-dimensional model of the city by ensuring the construction of a model of the city based on separately recognized city objects obtained from previously cleared laser scanning data.

The claimed result is achieved due to the implementation of the method of automatic construction of three-dimensional city models, containing stages in which:

- receive a set of primary data containing urban objects, and the data is at least the point cloud data obtained during laser scanning and geo-referenced photographic data;

- perform the reduction of isolated clouds of laser scanning points from a set of primary data into a single point cloud;

- determine the points in the aforementioned single cloud of points that characterize the horizontal surface of the earth on which urban objects are located;

- define clusters that characterize isolated groups of points that are interconnected;

- carry out the cleaning of the aforementioned single point cloud, during which the points of the laser scanning cloud that characterize non-stationary objects, objects characterized by one or more points that are not connected with the points of a single cloud characterizing the horizontal surface of the earth on which city objects are located, or clusters are removed , and extended groups of points of low density;

- the construction of normals to each of the points of the cleaned single point cloud is carried out;

- the construction of planes for sets of points of a point cloud is carried out;

- the construction of the terrain is carried out according to the built normals and the mentioned planes;

- perform sequential recognition of static urban objects based on data obtained from a cleaned single cloud of laser scanning data and photographic data;

- for each detected urban object, at least geographical coordinates and linear dimensions are determined;

- carry out the construction of a three-dimensional model of each of the detected mentioned urban objects;

- perform the combination of the above three-dimensional models into a single three-dimensional model of the city.

In a particular embodiment of the method, the laser scan data is data obtained during ground surveying, aerial surveying, or a combination thereof.

In another particular embodiment of the method, the primary data further comprises semantic data and / or panoramic photographs.

In another particular embodiment of the method, primary georeferenced photographic data is obtained using ground and / or air and / or space photography.

In another particular embodiment of the method, the semantic data contains metadata of urban objects.

In another particular embodiment of the method, at the stage of constructing the normals of the cloud points, the nearest neighboring points are determined for each cloud point.

In another particular embodiment of the method, for each point for which neighboring points are determined, planes are constructed by estimating model parameters based on random samples of RANSAC (Random sample consensus).

In another particular embodiment of the method, for the points characterizing the horizontal surface of the earth on which urban objects are located, the level of their location is determined.

In another particular embodiment of the method, at the stage of determining the clusters, clusters are first formed that characterize groups of connected points lying in a predetermined limit above the level of points that characterize the horizontal surface of the earth on which urban objects are located.

In another particular embodiment of the method, connected points lying below and above the mentioned limit are added to the formed clusters.

In another particular embodiment of the method, when combining point clouds, a comparison is made of the presence of clusters that are inherent to only one point cloud.

In another particular embodiment of the method, identified clusters that are inherent to only one point cloud are removed from a single point cloud at the stage of cleaning.

The claimed solution is also implemented due to the system of automatic construction of three-dimensional city models, containing at least one processor and at least one memory containing machine-readable instructions that, when executed by at least one processor, perform the above-described method for automatically building three-dimensional city models.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates the main steps performed in the implementation of the claimed method.

FIG. 2 illustrates the steps for performing a point cloud cleaning process.

FIG. 3 illustrates the steps for sequentially recognizing urban objects.

FIG. 4 illustrates a point cloud obtained by laser scanning.

FIG. 5 illustrates an example of constructing normals to cloud points.

FIG. 6 illustrates the definition of point clusters.

FIG. 7, 8 illustrate an example of determining points characterizing the surface location of urban objects.

FIG. 9 illustrates an example of the claimed system.

DETAILED DESCRIPTION OF THE INVENTION

The claimed system of automatic construction of city models (CIMP), as well as the method that implements it, is a software and hardware platform that allows, based on data obtained from various sources, for example, ground and / or airborne laser scanning, photo panning, Earth remote sensing data (remote sensing) , fully automatically build three-dimensional models of cities in a fairly short time. The resulting models can be exported to various GIS and CAD systems for their further display and modification.

A three-dimensional model of a city means a combination of urban objects (buildings, structures, fences, poles, etc.), terrain, and other entities mathematically described in generally accepted 3D formats (3DS, OBJ, etc.). Such a description for each object may contain geometry (sets of vertices, relations, normals, etc.) and materials (textures).

According to FIG. 1, when implementing the claimed method (100) for building a city model at step (101), primary (initial) data are obtained for subsequent processing. Point clouds (in generally accepted formats, las), geo-referenced photo images, and semantic information are used as initial data for constructing CIMP models. As photographic materials, ground-based photographing data, surveys from unmanned and manned aerial vehicles, satellites can be used. Semantic data may include the contours of houses, number of storeys, the available coordinates of objects, etc.

At step (102), data from various sources is combined (laser scanning, photo image, etc.). In particular, this procedure is aimed at the formation of a single cloud of points from isolated clouds obtained during laser scanning.

When shooting and laser scanning, both ground and air, the survey equipment uses professional GPS / GLONASS receivers that work together with a network of base stations, as well as inertial units. However, in a city, the positioning error can still be more than 1 meter. This leads to the appearance in the clouds of points of a significant echo from different driveways - the same object is represented by several similar sets of reflections. To refine the track, information methods are used for characteristic areas of space obtained from clouds of points or photographs. The most important task of compiling data from various sources is the calculation of such areas.

Such areas may be, for example, compact recognized objects, such as poles. Since the positioning error is rarely more than 3 meters, and the distance between real poles is almost always greater than this threshold, identification can be carried out. Since at each moment of time it is known with what azimuth and at what distance from the car the reflection from the column was obtained, using several reference points, you can adjust the track of the machine so that the selected reference objects, and therefore the whole scene, have the smallest shifts from different driveways.

Another potential information opportunity is provided by photographs. In automatic mode, photos from different driveways search for characteristic images. Further, since the photo is geo-referenced, it is possible to adjust the track for some such images with similar algorithms.

In addition to the initial data, for the analysis of territories from the air, combined georeferenced mosaics of aerial or satellite images in jpg / tiff formats with the coordinates of individual tiles can be used, it is also possible to use images in the GeoTIFF format. The spatial data (geodata) for recognition are represented by vector layers in Shapefile format containing coordinates (WGS84 coordinate system Web Mercator projection): the boundaries of buildings, roads, and other objects. Various semantic related information can also be used - metadata, which can be storeys, floor plans of buildings, etc.

At step (103), the points of a single point cloud (Fig. 4, pos. (200)) are determined, which characterize the surface of the location of urban objects or the so-called “land”. To increase the accuracy of determining urban objects, before starting their recognition algorithms, it is necessary to first divide all the points of a single point cloud into "ground" and clusters - groups of points connected to each other, but not connected to points of other clusters other than through the "ground".

Points characterizing the “earth” describe a horizontal surface in a cloud of laser scanning points on which urban objects stand, in particular, they can be a roadway, sidewalks, lawns, etc. The ground level (Fig. 7, 8) may vary, its the value is calculated as the minimum measurement height in this coordinate, averaged and processed by a median filter, to get rid of small holes and irregularities. To do this, the entire horizontal plane (which is specified by the XY coordinates) of the point cloud is divided into a fine grid - of the order of 10x10 cm. For each such cell, an attempt is made to determine the “ground” level. Among the points that fall into the 10x10 volumetric cell and are used to determine the minimum, approximately 2% of the lowest points are discarded to eliminate the influence of spurious interference, for example, points that are reflected in a puddle and thus gain a height of -2 meters relative to the true level of the urban surface objects ("land"). The minimum height of the remaining points is taken as the ground level in this cell. Next, the level of each cell is aligned with a median filter relative to the level of neighboring cells in a square of 70x70 (i.e. three in each direction). The ground level of this cell is the median among the levels of neighboring cells.

At step (104), cluster formation is performed. The initial clusters are formed at a given height, for example, 0.5-1.5 meters or another specified limit (0.3-1.3, 0.4-1.4, etc.), which characterizes the groups of connected points of a single point cloud. Points above and below are added to the originally constructed clusters later. This restriction allows you to get rid of the connectedness on the grass and other low objects, as well as on wires or tree branches. It is believed that two points belong to the same cluster if the distance between them is less than 20 cm. After determining the initial clusters, points lying above 1.5 and below 0.5 meters above the ground are added to them by the same principle. At the same time, points that do not belong to any of the already constructed clusters form new clusters. In FIG. 6 shows an example of building clusters (202) for objects from a single point cloud (200) shown in FIG. four.

Next, at step (105), the resulting single point cloud is cleaned. According to FIG. 2 first, the determination of non-stationary (moving) objects (step 1051), for example, cars, pedestrians, animals, etc. To speed up the processing of point clouds and reduce the likelihood of errors, the corresponding points characterizing these objects are deleted. To do this, after converging by comparing clouds from different driveways, clusters are selected that are visible only on one of them. Such clusters are recognized as moving and are removed from the clouds.

Also, simply individual points or sets of several points that are not connected in any way with points characterizing the surface of the location of urban objects or other clusters are deleted (step 1052). Other cleaning methods are also used, for example, according to geometric characteristics. So at step (1053), long sequences of single points above the carriageway belonging to the wires are deleted. These are connected sets of points with a thickness of 1-2 points and a length of units or tens of meters.

After that, at step (106), based on the data obtained during the cleaning of a single point cloud, sequential recognition of city objects is performed. The input data for this are the previously cleared single cloud of laser scanning points and other prepared data, such as panoramas, semantic data, and air and space photographing tiles. The output is geo-referenced models of individual objects with geometry and textures, road contours, and other individual entities.

According to FIG. 3, in order to perform recognition of urban objects in step (106), a series of sub-steps are performed. At the first sub-step (1061), normals to the points of the resulting single point cloud are constructed. According to FIG. 5 to each of the points of the cloud (200), normals (201) are built, the points in the plane (1062) are combined and the terrain (1063) is built based on aerial photography. To determine the normals (201) and planes (block 1062), the laser points are grouped by location and time of the survey. Further, for each point of the cloud (200), the nearest neighboring points are taken and the planes are constructed by the RANSAC method. The normal vector of the plane that passes through the largest number of neighboring points is considered the normal to this point.

To construct the planes (block 1062), the points of the cloud (200) are divided along a planar grid. In each cell of this grid, for each point and the normal corresponding to it, the neighboring points belonging to the plane described by this pair are searched. The point of the point cloud with the maximum number of neighboring points on the plane is taken as the basis for the plane, and all the mentioned neighboring points are removed from further processing, after which the process is repeated. By “neighbors in the plane” are meant points of a cloud that lie in a plane containing the current point 130 and its perpendicular normals. After all the planes are found, an attempt is made to combine these planes with planes from neighboring cells. The planes are combined only if the distance L between them is not greater than a given value, for example 20-30 centimeters. This principle allows you to divide among themselves houses, standing on the same line.

At the sub-step (1063), the terrain is constructed using photogrammetric algorithms based on images obtained during aerial photography. Additionally, at the aforementioned sub-step (1063), reliefs of individual objects can be constructed, for example, parts of a building (roof, facade, etc.).

Next, urban objects are determined (sub-step 1064). Objects of different types are searched sequentially in the processed source data, while those sets of points in the clouds or portions of images that have been correlated with any object are marked and do not participate in further processing. When implementing the method, scripts for detecting objects in the following sequence are automatically executed: buildings and structures, fences, poles, billboards, signs and traffic lights, bus stops, and other objects. The sequence can be changed manually or automatically for best recognition results. Separate parts of the process can also be removed or isolated, for example, you can configure the script to skip a specific type of object (poles, billboards, fences, etc.).

Each recognition script has default settings, according to which the platform performs automatic recognition. There is also the possibility of creating a settings template for each search algorithm, which can then be used by default.

To recognize a building based on cartographic and semantic data, the contour of the building and, possibly, its approximate height are determined. An array belonging to the desired object is extracted from the cleaned single cloud of points, and the necessary textures are selected from aerial photography. To eliminate discrepancies in the positioning of the object on the maps, in the cloud of points and panoramas, an intelligent alignment system is used, which allows you to “attract” the necessary data sections even with a shift of several meters.

The highlighted information is checked for a number of key features that identify the object as a building. Such signs are the presence of extended vertical flat surfaces along the boundaries of the object, the presence of window cavities in the clouds of points, the height of the object above the surrounding relief, the presence of a characteristic roof when shooting from the air and some others.

The fence is defined as an extended plane adjacent to the ground, with a height of no more than a given limit. A number of additional features can be used as data preventing the erroneous detection of third-party objects as fences: the absence in the space region of the contours of houses, as well as other adjacent objects, the absence of an additional relief on the plane, a uniform average texture color, etc.

In the general case, the algorithm for determining the pillars consists of three stages.

1. The selection of cylindrical vertically oriented objects.

2. Learning adaptive boosting algorithm.

3. Classification of found objects using the constructed algorithm.

At the first stage, the points are divided by a grid of square meters. For the points of each square and its neighbors, three neighboring points are randomly selected several times and a cylinder is constructed through these points. If in the grid element the centers of such cylinders lie in the same plane coordinate, this place is marked as a vertically oriented cylinder. For each such place, signs are constructed for the learning algorithm, namely

A) the radius and curvature of the surface;

B) the distribution of points away from the center of the cylinder;

C) distribution of points in the XY plane;

D) the distribution of points vertically;

D) the distance to the car track;

E) the average color of the points fixed on the three nearest panoramas;

G) the average reflection coefficient.

At the second stage, the adaptive boosting algorithm (ADABOOST) is provided with features of objects and classification. The algorithm builds a recognition cascade. In our case, two separate cascades are built - the first rough, sifting out 95% of the objects. The second is more accurate, it eliminates another 99%. This approach is used for further error correction - if with the help of this algorithm a missing object is found, it is most likely to be in the objects after the first stage. This reduces the number of viewed objects in general, without compromising the accuracy of object detection.

Traffic sign recognition occurs in the optical range. As auxiliary source data, a specially prepared database with a lot of real images of signs on the ground is used. The algorithm consists of constructing the characteristic features of the image and training the adaptive boosting cascade using examples from the existing character base. In point clouds, to eliminate errors of the second kind, secondary signs are controlled: linear dimensions, distance from the roadway, installation height, etc.

Traffic light recognition simultaneously takes place in a point cloud and ground photo panorama. A column of certain parameters with a characteristic traffic light box is detected in the cloud. In optical photography, circles are searched with the specified colors. To clip erroneous objects, cartographic data about the roadway and intersections are used.

A billboard is defined as a plane located at a certain height with certain linear dimensions. To eliminate errors of the second kind, objects-applicants are also checked for the absence of other objects located nearby, especially fences and buildings. Secondary signs are also checked, for example, the presence of a pillar as a support, the presence of an advertising texture with sharp drops in the middle color.

A public transport stop is detected by the presence of perpendicular planes of a certain size, visible from above the horizontal plane of the characteristic texture, as well as the presence of additional signs: the presence of an object “sign” of a certain type and a broken yellow band in the optical image. Also controlled are the distance from the roadway and a number of other factors.

By performing recognition of static urban objects at step (106), it is possible to recognize other objects that were not originally included in the recognition process. For this, this recognition method has the possibility of learning by analyzing a library of real images, as well as by setting several geometric parameters characterizing the size and position of objects. The training module is built according to an iterative scheme and it is possible to retrain the algorithms, indicating to it the errors of the first and second kinds after checking the recognition results of the next stage.

In the sub-step (1065), for each identified object in step (1064), at least the geographical coordinates in WGS84 (longitude, latitude and height) and linear dimensions are determined. Additionally, can be defined:

- the area of the planes of the object (for polygonal objects);

- type of object (with subtypes for some classes of objects, for example, road signs).

For the objects identified in step (106), polygonal three-dimensional models are constructed in step (107) containing textures in a graphic format, for example, JPG or JPEG, obtained from photographs.

The obtained spatial data can be exported to geographic files of vector formats SHAPEFILE, KML or GeoJSON for presentation in various GIS, in three-dimensional models in 3DS or OBJ formats for use in CAD systems and in other ways.

Next, at step (108), after recognizing and building models of urban objects, individual objects are combined into a single three-dimensional model, which, for the convenience of displaying and working with it, can be divided into separate fragments and recorded in several files. The output formats are standard and allow you to work with the model in third-party 3D graphics applications. The resulting data contains both the geometry of individual objects and their texture.

Before compiling the model, it is possible to specify a set of objects included there. So, for example, a model can be built only from buildings and structures with terrain or only from roads with road signs. This makes it easier to work with data in solving specific practical problems.

In FIG. 9 illustrates an example system (300) for implementing the inventive method (100). The claimed system (300) includes the following components.

Random access memory (RAM) (302), designed for online storage of instructions executed by one or more processors (301).

The storage medium (303) can be a hard disk (HDD), solid-state drive (SSD), flash memory (NAND-flash, EEPROM, Secure Digital, etc.), an optical disk (CD, DVD, Blue Ray) , mini-disk or their combination.

Input / output (I / O) interfaces (304) are standard ports and devices for pairing devices and transmitting data, selected based on the required configuration of the system (300), in particular USB (2.0, 3.0, USB-C, micro, mini ), Ethernet, PCI, AGP, COM, LPT, PS / 2, SATA, FireWire, Lightning, etc.

I / O facilities (305) are also selected from a well-known range of different devices, for example, a keyboard, touchpad, touch display, monitor, projector, pointing device, mouse, joystick, trackball, light pen, stylus, sound output devices (speakers, headphones, built-in speakers buzzer) etc.

Data transmission tools (306) are selected from devices designed to implement the communication process between different devices via wired and / or wireless communication, in particular, such devices can be a GSM modem, Wi-Fi transceiver, Bluetooth or BLE module, GPS module, Glonass module, NFC, Ethernet module, etc.

System components (300) are interfaced via a common data bus (307).

System (300) in the preferred embodiment is a server platform that provides the necessary calculations when implementing the aforementioned method of building a city model (100). In private versions of the system (300) can be implemented on the basis of mobile devices, such as a laptop, smartphone or tablet.

The information on the implementation of the claimed invention set forth in these materials of the application should not be construed as information limiting other, private embodiments of the claimed invention, not going beyond the disclosure of information in the materials presented, and which should be obvious to a person skilled in the art having the usual qualifications for which the claimed technical solution is designed.

Claims (25)

1. A method for automatically building three-dimensional models of cities, containing stages in which: receive a set of primary data containing urban objects, the data being at least the point cloud data obtained during laser scanning and geo-referenced photographic data; converging isolated clouds of laser scanning points from a set of primary data into a single point cloud; determining points in said single point cloud that characterize the horizontal surface of the earth on which urban objects are located; define clusters that characterize isolated groups of points that are interconnected; carry out the cleaning of the said single point cloud, during which the points of the laser scanning cloud characterizing non-stationary objects, objects characterized by one or more points that are not connected to the points of a single point cloud characterizing the horizontal surface of the earth on which the city objects are located, or clusters are removed , and extended groups of points of low density; building normals to each of the points of the cleared single cloud of points; construction of planes for sets of points of a point cloud is carried out; the terrain is built according to the built normals and the mentioned planes; perform sequential recognition of static urban objects based on data obtained from a cleaned single cloud of laser scanning data and photographic data; for each detected urban object, at least geographical coordinates and linear dimensions are determined; constructing a three-dimensional model of each of the detected mentioned urban objects; perform the combination of the above three-dimensional models into a single three-dimensional model of the city. 2. The method according to claim 1, characterized in that the laser scanning data are data obtained during ground-based surveys, aerial surveys, or a combination thereof. 3. The method according to claim 1, characterized in that the primary data additionally contains semantic data and / or panoramic photographs. 4. The method according to claim 1, characterized in that the primary geo-referenced photographic data is obtained using ground and / or air and / or space photography. 5. The method according to claim 3, characterized in that the semantic data contains metadata of urban objects. 6. The method according to claim 1, characterized in that at the stage of constructing the normals of the cloud points for each cloud point, the nearest neighboring points are determined. 7. The method according to claim 6, characterized in that for each point for which neighboring points are determined, planes are constructed by estimating model parameters based on random samples of RANSAC (Random sample consensus). 8. The method according to claim 1, characterized in that for points characterizing the horizontal surface of the earth on which urban objects are located, determine the level of their location. 9. The method according to claim 8, characterized in that at the stage of determining the clusters, clusters are first formed that characterize groups of connected points lying in a predetermined limit above the level of points characterizing the horizontal surface of the earth on which urban objects are located. 10. The method according to claim 9, characterized in that connected points lying below and above the aforementioned limit are added to the formed clusters. 11. The method according to claim 1, characterized in that when combining point clouds, a comparison is made of the presence of clusters that are inherent to only one point cloud. 12. The method according to claim 11, characterized in that the identified clusters that are inherent in only one point cloud are removed from a single point cloud at the stage of cleaning. 13. A system for automatically building three-dimensional city models, comprising at least one processor and at least one memory containing machine-readable instructions that, when executed by at least one processor, perform a method for automatically building three-dimensional city models according to any one of claims. 1-12.

Images