JP7418281B2 - Feature classification system, classification method and its program - Google Patents

Description

The present invention relates to a technology for classifying features using aerial images such as satellite images.

The technology behind the present invention will be explained. In the text below, extraction means recognizing the type without knowing what it is, and categorizing means recognizing after specifying the type.

Satellite images are suitable for investigating ground cover in the same area because they can cover a wide area in one shot. Research toward the practical use of reusable rockets, miniaturization of Earth observation satellites, and constellation plans are underway around the world, and the trend in satellite Earth observation is towards high resolution, low cost, and high frequency imaging. Against this backdrop, the number of satellite images that must be interpreted is expected to increase explosively, and there are concerns about a shortage of manpower to interpret them. Therefore, the demand for technology that automatically interprets satellite images is increasing year by year.

Many applications of deep learning have been reported in relation to image interpretation.
However, compared to ordinary photographs taken with a digital camera, it is difficult to produce highly accurate classification results from satellite images. There are several possible reasons for this, but the first is that satellite images are taken from directly above, and are extremely extreme images that basically only show one side from above. As a result, the top and bottom of the image have no meaning, and there is no information about typical layout patterns within the image, making it difficult to separate the foreground and background. Second, another factor that makes automatic interpretation difficult is that the conditions for capturing satellite images differ each time due to the weather, the positional relationship between the sun and the earth, and the like. Patent Document 1 provides a mechanism that supports the user's interpretation by simulating how a known 3D model looks depending on changes in imaging conditions. Third, there are various terrestrial features in satellite images, and there are too many variations in the labels that humans have given to the terrestrial features, making it difficult to apply machine learning. Labels given by humans can be difficult for computers to understand, such as classifying things that look different or have completely different sizes into the same class. For example, in an urban area, it is known that when a deep learning model is generated that extracts buildings with various numbers of pixels as the same class, the performance deteriorates.

When automatically interpreting satellite images, careful attention must be paid to resolution adjustment. Patent Document 2 discloses a method that can improve classification performance by deep learning by creating a multi-task network that trains three stages of size-based detectors when building a building classification network. By using common sense that people unconsciously use, such as dividing the image into three sizes, we are achieving certain results in tasks such as extracting buildings and road surfaces.

When classifying buildings and road surfaces, it is difficult to make indirect inferences using background knowledge and information outside the image range, so it is not as simple as the extraction task, but as mentioned above, it is difficult to classify buildings and road surfaces. The cost of acquiring satellite images is decreasing, and recognition using a large number of images is becoming an option.

JP2010-271845A Japanese Patent Application Publication No. 2019-175140

Although deep learning has higher recognition performance than other methods, it is not perfect for applications that want to classify buildings and road surfaces as having functions. In the case of satellite images, some kind of background knowledge is required to identify the type, and it is practically difficult to perform this end-to-end. In such a case, if deep learning is executed end-to-end, it will take a lot of effort to correct the results, and there is a problem that the total number of man-hours required for labeling cannot be reduced.

The present invention aims to reduce the number of work steps by assisting labeling in classifying features.

The present invention proposes a method that achieves both the extraction and classification of buildings by performing step-by-step recognition of each of buildings and road surfaces, and assists labeling while presenting explanations of factors that serve as judgment indicators.

An example of the "feature classification system" of the present invention for solving the above problems is as follows:
It is a classification system that classifies features using multiple aerial images, and includes a road surface polygon extraction unit that extracts road surface polygons by integrating the extraction results recognized at multiple resolutions, and multiple aerial image pairs. a building polygon extraction unit that extracts building polygons using three-dimensional information obtained by multi-view stereo and the road surface polygon; and a building polygon extraction unit that extracts image characteristics and geometric information of each polygon for classifying the building polygon and the road surface polygon. An attribute table creation unit stores in an attribute table and classifies building polygons using the attribute table, and further updates the attribute table by evaluating relationships with other polygons for polygons classified with high accuracy. , and a classification section that repeats classification processing.

According to the present invention, the number of man-hours can be reduced by assisting labeling in classifying features.

Problems, configurations, and effects other than those described above will be made clear by the following description of the embodiments.

The overall processing flow diagram of the feature classification method. Functional block diagram of the feature classification system. Flow diagram for extracting building polygons and road surface polygons. A flow diagram for obtaining shaped polygons. Flow diagram for creating a building attribute table. A flowchart for creating a road surface attribute table. The figure which shows an example of a building attribute table. The figure which shows an example of a road surface attribute table. Flowchart of classification processing for buildings and road surfaces. The figure which shows the selection screen of the extracted edge. The figure which shows the result display screen of classified polygons. FIG. 2 is a diagram illustrating the hardware configuration of the present invention.

Embodiments of the present invention will be described with reference to the drawings.

An embodiment for classifying terrestrial features from satellite images will be described. In this embodiment, building areas and road surface areas are extracted from satellite images taken of the ground, and the building areas are categorized into, for example, schools, hospitals, shopping malls, etc., and the road surface areas are categorized into, for example, parking lots, runways, etc. , roads, etc.

FIG. 1A shows a flowchart of the overall processing of the feature classification method. In S101, building polygons (building areas) and road surface polygons (road surface areas) are extracted from the satellite image. Then, in S102, the building polygon and the road surface polygon are shaped to obtain a shaped polygon. Then, in S103, attribute values of the building polygon and road surface polygon are extracted to create an attribute table. Then, in S104, each polygon is classified using the attribute table.

Extraction processing of building polygons and road surface polygons will be explained with reference to FIG. 2, and processing of shaping the extracted building polygons and road surface polygons will be explained with reference to FIG. Creation of the building attribute table will be explained in FIG. 4, and creation of the road surface attribute table will be explained in FIG. 5. The classification process for buildings and road surfaces will be explained with reference to FIG.

FIG. 1B shows a functional block configuration diagram of a feature classification system for performing the processing flow of FIG. 1A. The classification system includes an image input section 11 that imports satellite images, a road surface polygon extraction section 12 that extracts road surface polygons from the satellite image, a building polygon extraction section 13 that extracts building polygons from the satellite image using the road surface polygons, and an extracted building. A shaped polygon acquisition section 14 obtains shaped polygons from polygons and road surface polygons, an attribute table creation section 15 creates attribute tables 17 for buildings and road surfaces from each shaped polygon, and classifies buildings and road surfaces based on the attribute tables 17. A classification section 16 is provided. It also includes a user interface 18 that displays a selection screen and performs input from the user as necessary.

Satellite images are images in which each pixel has real distance information, and the higher the spatial resolution of the image, the larger the image size becomes. When automatically interpreting images using software, if the image size is too large, the computational power often cannot keep up. Especially in deep learning, large-scale calculations are achieved through massively parallel calculations using GPUs (Graphics Processor Units), but as the image size increases, the neural network scale becomes extremely large, making calculations difficult. There's a problem. For this reason, the image resolution is generally adjusted before inputting it to the GPU, but as a result of image reduction, it is possible that features with originally small pixel sizes cannot be detected. Therefore, the basic approach is to prepare learning data that matches the pixel size of the feature to be recognized after determining the scale of the image. At this time, it may be difficult to determine the scale if the size of the feature you want to recognize is widely distributed from small to large, or if the size distribution itself is unknown. In such cases, approaches to detecting building areas using approaches other than deep learning may be effective. For example, when recognizing tall man-made objects such as buildings in an urban area, another approach is to use photogrammetry to create a three-dimensional image and extract the building using the height difference from what is determined to be the road surface. It can be considered as It is best to treat the three-dimensional data as a two-dimensional image called a DSM (Digital Surface Model) image, where each pixel represents the altitude of a feature.

As mentioned above, in satellite images, each pixel has distance information, so a 3D point group can obtain absolute xyz coordinates. yx is latitude and longitude, and z is highly convertible. It is recommended to draw contour lines on the DSM image and apply processing to treat areas where the height difference from the area extracted as the road surface is equal to or greater than a threshold as artificial buildings. Note that it is preferable to use expansion and compression processing to deal with small pixel omissions in the DSM image.

What is important when converting images into three-dimensional images is that the positional relationship between the subject and camera is not biased. It is best to set up a shooting plan with variations in the satellite shooting position and angle so that the entire surface of the building is photographed. However, since satellite images are not taken frequently, it tends to take a long time to collect the number of satellite images that can be converted into three-dimensional images. If the collection time becomes long, new structures may be built on the ground or natural objects may be leveled. In order to solve this problem, these effects can be reduced by using satellite constellations that take frequent photographs or by using satellites equipped with a mode that allows video recording of the same location.

If it is not possible to obtain an image set of sufficient quality to convert satellite images into three-dimensional images, that is, if there are no variations in the shooting conditions as described above, or if the image collection period becomes long, , the possibility of missing pixels in the DSM image increases. The 3D conversion method assumes Multi View Stereo, which integrates 3D point clouds obtained by stereo viewing from pairs of satellite images. It is best to integrate 3D point clouds by focusing on pairs of satellite images that are as new as possible. Specifically, when a 3D point cloud created by a pair of new satellite images and a 3D point cloud created by a pair of old satellite images are superimposed on the same geographic coordinates, the information of the former is It's good to prioritize. For example, one possible method is to calculate the z-coordinate values of the same point for a plurality of stereo pairs and average them, but the newer the stereo pair of satellite images, the greater the weight during averaging. Alternatively, using a stereo pair of new satellite images as a reference, the degree of deviance of the z-coordinate of a pair including a satellite image that has passed a predetermined number of years is determined, and if the degree of deviation is high, it may be excluded from averaging.

DSM images obtained in this way may have an overall slope of the ground, which may not be compatible with the process of detecting buildings based on the height difference from the road surface, so DSM images may not be compatible with DEM (Digital Elevation Model). It is recommended to use raster data representing the height of the ground surface to cancel out the slope. DEM does not include features that exist above the ground surface, so only information about the height of features above the ground surface can be calculated.

When detecting the road surface, extraction errors may occur due to shadows, occlusion of buildings, etc., and the correct area may not be extracted. In that case, it is preferable to recognize the same point using a plurality of images under different shooting conditions and integrate the results. For example, the influence of building occlusion can be excluded by using images taken at different angles, or the influence of shadows can be excluded by using images taken at different times. If the ratio of the road surface pixels that you want to recognize is too large compared to the image size that can be input to deep learning, the image cannot be used as shape information and accuracy will deteriorate significantly. Therefore, if the width of the road surface that you want to recognize is larger than the input image size, you can zoom out to recognize a wider area.In other words, the input image size is fixed and the resolution is downsampled to create an image that shows a wider area. Alternatively, it is preferable to prepare multiple stages and perform recognition on each stage. It is preferable to superimpose recognition results at these different resolutions and perform segmentation using the obtained information likelihood. It is a good idea to cut the resolution in half and stop zooming out when the target road surface width and input image size fall below a certain value.

FIG. 2 shows a flowchart for extracting building polygons and road surface polygons. First, in S201, processing is performed to exclude natural objects such as vegetation such as plants and water bodies from the image. In the road surface extraction process, images of multiple resolutions are obtained in the multi-resolution process of S206. Then, in S207, the road surface is extracted from the images of each resolution, the extraction results of the road surface of each resolution are integrated in S208, and the accurate road surface is extracted in S209. In the building extraction process, three-dimensionalization is performed in S202, a DSM image is created in S203, tilt correction is performed in S204, and denoising processing is performed to remove outliers from the three-dimensionalization process in S205. Then, in S210, building polygons are extracted using also road surface polygons.

The building polygons and road surface polygons are extracted by the building polygon extraction section 13 and the road surface polygon extraction section 12 in the block diagram of FIG. 1B.

As described above, polygons extracted from buildings and road surfaces are expected to contain a lot of noise, so as post-processing, the boundaries of the polygons are refined to obtain shaped polygons. That is, the shaped polygons are acquired in S102 of the processing flow in FIG. 1A. The shaped polygons are acquired by the shaped polygon acquisition unit 14 shown in the block diagram of FIG. 1B.

The process of extracting a shape desired by the user from the extracted polygons will be explained using FIG. 3. The polygon refining method of this embodiment is realized by a building polygon refining program, a road surface polygon refining program, and a user interface that can display the progress of these programs, thereby enabling interactive polygon extraction. Rather than requiring the user to specify each point, it is preferable to display the progress in progress and provide a mechanism that preferentially accepts changes only when the user selects a modification.

To extract buildings, it is assumed that DSM images will be utilized by converting the point cloud obtained by three-dimensionalizing it into three dimensions as described above, with the latitude and longitude on the xy plane and z as the height direction, but the extraction results It is natural to want to display it superimposed on the satellite image. This satellite image to be superimposed will be referred to as a reference image. However, when polygons extracted from a DSM image are superimposed on a reference image, the building may appear shifted from its original position. This is because, unless the satellite image is taken directly below, when a high-rise building is imaged, it is imaged so as to overlap points with different latitudes and longitudes. In order to avoid this influence, it is desirable to select a satellite image taken directly below as a reference image as much as possible (S311), but in principle it is difficult to completely eliminate this positional shift.

Therefore, this embodiment provides a method of improving the accuracy of extracted polygons using two-dimensional image features. First, representative points of each building polygon extracted from the DSM image are selected (S312). The method for selecting the representative point may be specified by the user, or the center of gravity of the building polygon, the peak of the height, etc. may be selected. It is preferable to cut out the reference image around these coordinates, cut out each chip image, and analyze the edges. The chip image is cut out so that the entire building polygon is captured, and since there is a possibility of connection with adjacent polygons, if the distance between the representative points of the polygons is closer than the specified value, the entire image of multiple polygons will be captured. It is a good idea to cut out the chip image like this.

Once the representative point is selected, the area around the image is cut out and edge detection processing is applied (S313). Hough transform, Canny filter, etc. are candidates for edge detection. Since the detection target is a building, it is likely to be rectangular, and by taking a directional histogram of edges, it is often possible to observe a tendency for the frequency to concentrate at angles that are 90 degrees apart. By utilizing this property, edges to be emphasized and edges to be excluded as noise can be specified. Once the edge of interest is determined, an approximate straight line is extracted. It is best to form a grid using approximate straight lines and select which grid to leave. It is recommended to superimpose the grid on a binary image obtained by rasterizing building polygons, and select it if a certain number of building areas are included, or to evaluate the continuity of the area using a graph cut. (S314). It is preferable to prepare an interface that allows the user to select candidates to keep from among the straight lines obtained in this way. It is preferable to prepare an interface that allows a method for selecting a straight line to be deleted from the displayed candidates, a method for selecting a straight line to be used, or a method for specifying a new straight line (S302). This can be expected to have the effect of reducing rework of annotation work and reducing labor. For example, it is preferable to install an interface that supports an edge candidate selection screen as shown in FIG. 9(a). In this way, the shape of the polygon is determined by connecting the points where the straight lines intersect, but there are cases where the original polygon is divided or combined (S315). At this time, it is preferable to display a result confirmation screen and a screen that allows correction as shown in FIG. 9(b) (S303). It is preferable to enable the user to cancel division and combination of polygons, manipulate the vertices of polygons, etc. on the screen for modifying the results. Vertex operations include operations such as moving vertices, adding new vertices, and deleting vertices. After checking and correcting the results, the building polygon is determined (S316).

The same flow applies to road surface polygons, but it is assumed that the polygon size of road surfaces is larger than that of buildings. If the resolution is left unchanged, the edge detection range will become wider, so it is better to perform processing at multiple resolutions, as in the case of segmentation (S321 to S326).

In addition to this method, a generative adversarial network that infers the contours of buildings and road surfaces is used as input data for building polygons extracted from the three-dimensional shape and chip images of satellite images. There is a way to sharpen the image. In this case as well, it is a good idea to prepare a screen for checking and correcting the results, and to set it so that you can manipulate the vertices of polygons.

Through the above flow, contour polygons of buildings and road surfaces are obtained.

In this embodiment, information for classifying the classes of the building polygons and road surface polygons obtained in this way is extracted. That is, building and road attribute tables are created in the processing flow of FIG. 1A (S103). Creation of the attribute table is performed by the attribute table creation unit 15 shown in the block diagram of FIG. 1B. FIG. 4 shows a flow for obtaining attribute values for categorizing building polygons, and FIG. 5 shows a flow for obtaining attribute values for categorizing road surface polygons. An ID is assigned to each building polygon to create a building attribute table as shown in FIG. 6, and an ID is assigned to each road surface polygon to create a road surface attribute table as shown in FIG.

Figure 4 shows the flow of creating a building attribute table. First, the height is extracted in S401. Since height information can be obtained from DSM images, it is a good idea to retain information on peak, median, and average values. Furthermore, the area is calculated in S402. The polygon area information is also useful information for the type of building, so it is calculated and stored as a column. Spectral features are calculated in S403. When calculating attribute values based on pixel values, such as polygon color information, texture, NDVI (vegetation index)/PNVI, it is recommended to use a range of about 5 to 9 pixels from the center of gravity. It is preferable to use pixel average values for color information and NDVI/PNVI. PNVI cannot be calculated unless the Red Edge spectrum is obtained, so it may not be possible to calculate it depending on the satellite image used, so if it cannot be calculated, it can be left unused. For the texture, it is preferable to use a gray level co-occurrence matrix (GLCM) and maintain parameters such as contrast, correlation, energy, and uniformity. In addition, the shape is extracted in S404. Classify polygon shapes into several patterns. For example, if the shape is square, which is most common in buildings, then 0, if the shape is circular, which is often seen in special structures such as storage warehouses, 1, and for other polygons, 2, 3, etc. It is a good idea to assign classes according to the purpose. In S405, characters and signatures are extracted. Facilities such as schools and police stations in Japan often have their names written on the roof, so a combination of one or more kanji such as "xx hospital", "xx police", "xx elementary school", "xx junior high", etc. It is a good idea to perform pattern recognition on , and store the detected ones in an attribute table. Other buildings, especially elementary and junior high schools, have numbers written on their roofs to make it easier for helicopters to spot them from the air in the event of a disaster. In other words, it is thought that the numbers on the roof can be used to identify the building, so if a number is included in the polygon, the pattern is detected and the found number is stored in the attribute table. It's a good idea to leave it there. The H-in-a-circle mark, which is a sign for a helicopter to land, can exist on both roads and buildings, but it also serves as reference information when specifying a building, so if it can be detected, it is retained in the attribute table. It's a good idea to keep it.

We pointed out in the assignment that it is difficult to use end-to-end classification by deep learning as is, but since the classification probability of each class obtained in the output layer of the neural network contains useful information, May be used as table value. As shown in S406 to S409, a chip image is generated centering on the building polygon (S406), and a class classification and object recognition network is applied to obtain the position, shape, classification probability, etc. of the classification class (S407). For example, a method of inferring the class of a chip image using a class classification network can be considered. When using a class classification network, the class and probability of the building in the chip image are output, so it is recommended to save the classification probabilities of the three highest probability classes in the attribute table (S409). .

On the other hand, when using an object recognition network, information on the position and class of the object in the chip image can be obtained. In satellite image recognition, there are only a few types of buildings that can be directly recognized, and it is assumed that this will be used to extract characteristic shapes. For example, oil storage facilities and radars look like white circles when photographed from above. Object detection networks are effective for extracting buildings with such simple shapes. When a building is extracted using an object detection network, the overlap between the extraction position and the building polygon is evaluated. The degree of overlap is evaluated using IoU (Intersection over Union) between the extracted building position/shape and the building polygon (S408). When the IoU value is above a certain level, that is, when the overlap is sufficient, it can be determined that the building detected by the object detection network and the building in the building polygon are the same. When using an object recognition network, position and shape information can be obtained, but when using the output from class classification, GradCAM etc. can be applied to obtain the area and IoU that served as the judgment index for each probability. It is better to take

Since the building polygon is generated based on the three-dimensional shape, the building polygon can be assumed to be completely orthogonal, but the chip image is assumed to be an image cut from a multispectral satellite image based on the coordinates of the building polygon. Therefore, depending on the imaging conditions, it may not be possible to make an orthogonal assumption. Therefore, it is a good idea to have an index for determining identity other than IoU. For example, if the distance between the positional shape detected by the object detection network and the building polygon is less than or equal to a predetermined value, it may be determined that they are the same. There are several possible definitions of distance, such as the distance between the center of gravity of a building polygon and the center of gravity of a position shape detected by an object detection network, or the shortest distance between both polygons.

Information on the probability of building classes that can be recognized by the object detection network is stored in a table with the classes with higher probabilities and their probabilities, similar to when the class classification network is used. It is also possible to perform simple shape extraction using an object detection network. For example, an object detection network trained to detect circular buildings can be used to search for circular regions in images. By combining this with the above-described identity determination process, it is possible to fill in not only the class and probability information columns of the attribute table but also the shape information columns.

The object detection network exhibits good performance not only in detecting the characteristically shaped buildings described above but also in detecting moving objects. Therefore, specific objects around the building may be detected, counted, and used as attribute values. For example, if a large number of cars are parked, it is likely to be a parking lot, if there are a lot of trains, it is likely to be an outdoor garage, if there are a lot of planes, it is likely to be an apron, and if containers are loaded, it is likely to be a warehouse. It serves as a material for classifying nearby buildings. Therefore, for each building polygon, the number of these important moving objects within a predetermined range is counted and stored in the attribute table (S410). When counting the number of cars, trains, airplanes, etc., even if they are not present in an image at a certain point in time, they often exist when looking at another image taken of the same location. Therefore, when counting these moving objects, it is preferable to use a plurality of satellite images, accumulate the number of detected moving objects at many past points, and take the average. Additionally, the distance between the feature and the feature that is the key for the classification is measured and stored in the attribute table. Key features vary depending on the area of interest. For example, at an airport, by measuring the distance from the runway, it becomes possible to classify various facilities such as terminals and instrument landing systems. Further, it is preferable to hold the number of building polygons and the number of moving objects included between key features in the attribute table.

FIG. 6 shows an example of a building attribute table created in the building attribute table creation flow of FIG. 4. The ID assigned to each building polygon includes the color spectrum value, texture, vegetation index, height, area, shape class, number of objects within a given distance, distance to the key, number of features between the key, It has items such as estimated classes and their probabilities based on machine learning, and extracted characters.

FIG. 5 shows the flow of creating a road surface attribute table. It is assumed that the road surface extracted at the time of the processing in FIG. 2 is classified by material, for example, area polygons of concrete and asphalt are assumed. In classifying the road surface class, detection of related objects placed on the road surface is an important process (S501). For example, a straight, narrow road surface with cars on it is a road, and a wide road surface with a large number of cars densely packed together is likely to be a parking lot. This process preferably uses the flow of the object detection network used when filling the attribute table of building polygons. Expand the area of the detected related object with a buffer of a certain size. Then, the overlapping range with the original road surface area polygon is determined, and the shape and coordinates of the area polygon are updated (S502). Extraction of related objects includes important information not only for updating the shape but also for classification. If a feature with a means of transportation is listed, it becomes easier to classify the type of road surface. If there are a lot of cars parked there, it's a parking lot, if there are a lot of trains, it's an outdoor garage, and if there are a lot of planes, it's an apron. Therefore, it is preferable that each road surface polygon retains information about the features placed on it.

Also, it is a good idea to divide the area around the area polygon into chip images, introduce machine learning that can obtain the classification probabilities of position, shape, and classification class, and save the results in a table (as with building polygons). S503-S505). For example, a deep learning instance recognition network may be used to extract the shape of the region and the classification class information. Instance recognition allows the extraction of road surface shape information in pixel units, so it is determined that the shape matches the initial road surface area polygon, the polygon is divided to cut out the overlapping area, and the shape and coordinates of the road surface area polygon are updated. (S506). Next, the same processing as that for the building polygons is performed. That is, the road surface attribute table is updated by extracting spectral features such as color, texture, and vegetation index obtained from the pixel values of the satellite image, shape extraction, area calculation, character extraction, etc. (S507 to S510).

Also, since the type of feature adjacent to the road surface polygon is important information, the contact between the building and the road surface polygon is determined. Information on polygons in contact is saved together with information on inclusion and adjacency (S511).

FIG. 7 shows an example of a road surface attribute table created using the road surface attribute table creation flow shown in FIG. The ID assigned to each road surface polygon includes the estimated material, color spectrum value, texture, vegetation index, character string, shape class, area, loading feature, adjacent building polygons, included building polygons, and the estimated class by machine learning. It includes items such as the probability.

After creating an attribute table for each polygon, classification processing for each polygon is performed in the processing flow of FIG. 1A (S104). The classification process is performed by the classification unit 16 in the block diagram of FIG. 1B.

FIG. 8 shows a processing flow of classification processing. The classification processing method of this embodiment is realized by a building classification program, a road surface classification program, and a user interface that can display the progress of these programs, and enables interactive classification processing.

Buildings with specific functions in society are supposed to perform important functions at the location where they are installed. Therefore, the location has meaning and can be used to identify surrounding features. Conversely, when the location of a specific feature is known, the number of features that can be categorized may increase in a chain. It is advisable to set the degree of association between these classification classes based on rules, or calculate the degree of association in the training phase of machine learning. Road surfaces may have different names depending on the surrounding buildings even if they have a similar appearance, so in most cases it is possible to classify the road surface after the buildings have been classified.

First, it is preferable to classify the classes of features using machine learning or rule-based methods based on the values in the attribute table for each polygon. At this time, information on the attribute values used in the determination and the level of accuracy are retained.

A shortcut may be provided that allows the user to enter a key feature in advance via the user interface (S801). Once the information on the features is determined, the information necessary for categorization is gathered, and the number of buildings that can be categorized increases. When the key is determined, the attribute value (attribute table) is updated (S802). For example, in the example of the building attribute table in FIG. 6, the distance to the key and the number of features between the key and the key are added, and the estimated class is updated. Then, class evaluation is performed for each polygon (S803), and among the information used for the determination, polygons showing results with the highest accuracy are presented on the user interface in order (S804, S805). It is preferable to display all the classifications that have been classified with an accuracy exceeding a certain threshold value. Category candidates are confirmed and input through the user interface (S806). Prepare a mechanism to receive the correctness of the classification results on the user interface, update the attribute values based on that information, repeat the classification process and display on the user interface, and gradually increase the number of classified features. go. This iterative process ends when there are no features whose accuracy exceeds a certain threshold. Then, the information on the class that was not found and the updated attribute table are sent to the user interface (S807), the reference information and decision basis are updated on the user interface (S808), and the process ends.

In this embodiment, the interface for displaying the classification results presents the most likely classification class and displays the attribute values that are the basis when each polygon is selected, as shown in FIG. 10, for example. In the example of FIG. 10, "Hospital (score: 0.93), XX hospital text, helipad available, university nearby" is displayed. It is also good to have a mechanism to visualize the positional relationship with other terrestrial objects used during detection. For example, if there is an extremely large number of cars parked near a port, there should be a system in place to visualize the relationship between features that serve as association conditions, such as if the adjacent building is a trade-related facility. Good. In the example of FIG. 10, "University (score: 0.63), XX school text, pool available, hospital nearby" is displayed. When a polygon classification result is selected, in addition to displaying the classification class and attribute value, it is a good idea to link related polygons with lines, display a pop-up, or make the polygon blink. After evaluating all building and road polygons, if there are any features that are closely related to the features classified in the area but have not been discovered, consider past cases as a reference and find the missing features. It is advisable to present a report together with the attribute information that has been included (S807, S808). The report screen may be displayed in a separate window from the category candidate presentation screen, in a separate frame or tab within the same window, or may be output as a log.

FIG. 11 shows a hardware configuration for realizing the present invention. The CPU 53 starts the program of the present invention and executes satellite image classification processing. Image data used for calculations, such as satellite images, is temporarily stored in the memory 55, but if high-speed calculations are required, the data may be transferred to the memory of the GPU 54 and the calculation support of the GPU may be used. The classification results are displayed on the display device 51. The user checks the validity based on the displayed results and inputs the evaluation results through the user interface 56. Upon receiving the input of the evaluation results, the attribute table on the storage device 52 is updated. The communication interface 57 may be used when configuring a system with multiple computers or to execute actual calculation processing on an instance on the cloud.

The feature classification system of the present invention can be realized in a computer by having a CPU load a predetermined program onto a memory, and by having the CPU execute the predetermined program loaded onto the memory. This predetermined program may be inputted from a storage device in which the program is stored or from a network via a communication device and loaded directly onto the memory, or once stored in an external storage device and then Just load it into memory.

The invention of the program in the present invention is thus a program that is incorporated into a computer and causes the computer to operate as a feature classification system. By incorporating the program of the present invention into a computer, a feature classification system shown in the functional block diagram of FIG. 1B is configured.

Although the above embodiments have been described using satellite images as an example, the present invention is not limited to satellite images, but can also be used for general aerial images taken of the ground from above by an aircraft or the like.

According to the present embodiment, the number of man-hours can be reduced by assisting labeling in classifying features.

When a large amount of aerial images become available, it is possible to provide a method for classifying features in aerial images that incorporates three-dimensional information.

11 Image input section 12 Road surface polygon extraction section 13 Building polygon extraction section 14 Shaped polygon acquisition section 15 Attribute table creation section 16 Classification section 17 Attribute table 18 User interface 51 Display device 52 Storage device 53 CPU
54 GPUs
55 Memory 56 User Interface 57 Communication Interface

Claims (15)

A classification system that classifies features using multiple aerial images,
a road surface polygon extraction unit that extracts road surface polygons by integrating extraction results recognized at multiple resolutions;
a building polygon extraction unit that extracts a building polygon using three-dimensional information obtained by multi-view stereo using a plurality of pairs of aerial images and the road surface polygon;
an attribute table creation unit that extracts image features and geometric information of each polygon for categorizing the building polygons and the road surface polygons and stores them in an attribute table;
a classification unit that classifies building polygons using the attribute table, further evaluates relationships with other polygons for polygons classified with high accuracy, adds and updates the attribute table, and repeats the classification process;
A feature classification system with The feature classification system according to claim 1, further comprising:
For the small polygon groups that make up the extracted building polygons and road surface polygons, a certain rectangular area is set to include each small polygon, the aerial photographed image is cut out, edges are extracted, and edge information is obtained. A system for classifying features, comprising: a shaped polygon acquisition unit that obtains shaped polygons using a shaped polygon acquisition unit. The feature classification system according to claim 2,
The shaped polygon acquisition unit takes an angle histogram of edges, forms a straight line grid using the accuracy with the highest frequency, superimposes the straight line grid on data obtained by binary rasterizing the small polygon, and A classification system for features characterized by selecting a grid to be left with an occupancy ratio of , and obtaining shaped polygons. The feature classification system according to claim 1,
The attribute table is
a building attribute table having attribute values of the building polygon;
a road surface attribute table having attribute values of the road surface polygon;
A classification system for features characterized by consisting of: The feature classification system according to claim 1,
The attribute table assigns an ID to each polygon, and includes geometric information such as area, height, and shape of each polygon, image features such as color, vegetation index, and texture obtained by image processing, and estimated classes of classification results by deep learning. , a feature classification system characterized by including a positional relationship of feature polygons obtained by the classification section. The feature classification system according to claim 2,
has a user interface,
The feature classification system is characterized in that the user interface displays in stages the shape of the shaped polygon obtained by the shaped polygon acquisition unit, and accepts vertex editing and line correction by user operations. The feature classification system according to claim 1,
has a user interface,
A system for classifying features, wherein the user interface displays the classification results obtained by the classification section, and accepts additions, deletions, and modifications of the classification results by user operations. The feature classification system according to claim 1,
has a user interface,
The feature classification system is characterized in that the user interface displays the classification result obtained by the classification section together with one or more attribute values serving as a basis. The feature classification system according to claim 1,
has a user interface,
When the classification result for each polygon obtained in the classification section is calculated using the positional relationship with other polygons, the user interface adjusts the relationship by adding straight lines connecting the polygons or making the polygons blink. A classification system for features that emphasizes gender. A classification method for classifying features using multiple aerial images, the method comprising:
a road surface polygon extraction step for extracting road surface polygons by integrating extraction results recognized at multiple resolutions;
a building polygon extraction step of extracting a building polygon using three-dimensional information obtained by multi-view stereo using a plurality of aerial image pairs and the road surface polygon;
an attribute table creation step of extracting image features and geometric information of each polygon for categorizing the building polygons and the road surface polygons and storing them in an attribute table;
a classification processing step of categorizing buildings using the attribute table, further evaluating relationships with other polygons for polygons that have been classified with high accuracy, adding and updating the attribute table, and repeating the categorization process;
A classification method for features with The method for classifying features according to claim 10, further comprising:
For the small polygon groups that make up the extracted building polygons and road surface polygons, a certain rectangular area is set to include each small polygon, the aerial photographed image is cut out, edges are extracted, and edge information is obtained. A method for classifying features characterized by comprising a step of obtaining a shaped polygon using a shaped polygon. The method for classifying features according to claim 11,
The shaped polygon obtaining step includes:
Cutting out the periphery of the polygon before shaping from the aerial photographed image, taking an angle histogram of the edge, forming a straight line grid using the accuracy with the highest frequency, and superimposing the straight line grid on the data obtained by binary rasterizing the small polygon, A method for classifying features characterized by selecting a grid to be left with an occupancy ratio of 1 within the grid and obtaining shaped polygons. 11. The method for classifying features according to claim 10,
A method for classifying features, wherein the attribute table includes a building attribute table having attribute values of the building polygons and a road surface attribute table having attribute values of the road surface polygons. The method for classifying features according to claim 11 ,
The attribute table assigns an ID to each polygon obtained in the shaped polygon acquisition step, and includes geometric information such as area, height, shape, etc., and image characteristics such as color, vegetation index, texture, etc. obtained by image processing. , an estimated class of classification results obtained by machine learning, and a positional relationship of feature polygons obtained in the classification processing step.
to the computer,
A classification method for classifying features using multiple aerial images, the method comprising:
a road surface polygon extraction step for extracting road surface polygons by integrating extraction results recognized at multiple resolutions;
a building polygon extraction step of extracting a building polygon using three-dimensional information obtained by multi-view stereo using a plurality of aerial image pairs and the road surface polygon;
For the small polygon groups that make up the extracted building polygons and road surface polygons, a certain rectangular area is set to include each small polygon, the aerial photographed image is cut out, edges are extracted, and edge information is obtained. a shaped polygon acquisition step of obtaining shaped polygons using
an attribute table creation step of extracting image features and geometric information for classifying the building polygons and the road surface polygons and storing them in an attribute table;
a classification processing step of categorizing buildings using the attribute table, further evaluating relationships with other polygons for polygons classified with high accuracy, adding and updating the attribute table, and repeating the categorization process;
A program to run.

Images