JP7096034B2 - Building extraction system - Google Patents

Description

The present invention relates to a building extraction system.

Techniques for extracting buildings from data such as images acquired from the sky, such as aerial photographs and satellite images, are being researched. Patent Document 1 discloses a system in which a work area including a building that a worker wants to extract is specified on an image such as an aerial photograph, and the outline of the building is automatically extracted in the work area. Patent Document 2 below discloses an apparatus for extracting the contour of a building from the sky using a DSM (Digital Surface Model) acquired from the sky using a laser scanner or the like.

Patent Document 3 discloses an ensemble detector having three scales in an object detection device for recognizing a pedestrian, and discloses that the size of an image of a pedestrian to be detected differs depending on the scale. There is.

Japanese Unexamined Patent Publication No. 2011-76178 Japanese Unexamined Patent Publication No. 2013-101428 Japanese Unexamined Patent Publication No. 2018-5520

The inventors are developing a method of extracting a building using a convolutional neural network, for example, in order to reduce the workload of detecting a change (new construction or demolition) of a building. When extracting a building using a convolutional neural network, it was difficult to suppress oversight in the extraction of the building.

The present invention has been made in view of the above problems, and an object of the present invention is to provide a building extraction system capable of suppressing oversight in the extraction of buildings.

(1) For a plurality of buildings whose areas belong to the first range, the first learning input image having the first scale and the shapes of the plurality of buildings included in the first learning input image are shown. A second learning device having a second scale for a first building detector learned using informational teacher data and a plurality of buildings belonging to a second range whose area is different from the first range. A second building detector trained using the input image and teacher data including information indicating the shapes of a plurality of buildings included in the second learning input image, and a learning target area on the ground surface are in the sky. The feature information of the first input image taken from the above is input to the first building detector, and the first input image is enlarged or enlarged according to the ratio of the first scale to the second scale. An input unit that inputs the feature information of the reduced second input image to the second building detector, an output of the first building detector with respect to the feature information of the first input image, and the second. A building extraction system that includes an integrated unit that integrates the output of a second building detector with respect to the feature information of the input image of.

(2) In (1), the second scale is a building extraction system different from the first scale.

(3) In (2), the maximum value of the first range is larger than the maximum value of the second range.
The first scale is a building extraction system smaller than the second scale.

(4) In any of (1) to (3), the building included in the output of the first building detector and the building included in the output of the second building detector are removed based on the area. A building extraction system that further includes filters to do.

(5) In any of (1) to (4), the integrated unit has the output of the first building detector with respect to the feature information of the first input image and the feature information of the second input image. A process of enlarging or reducing at least one of the two outputs is executed so that the scale of the output of the second building detector and the output of the second building detector are the same, and the two outputs for which the process is executed are superimposed. Building extraction system.

(6) In any of (1) to (5), a third learning input having a first candidate scale for a plurality of buildings belonging to any one of the first range and the second range. A first candidate detector trained using the image and teacher data of information indicating the shape of the plurality of buildings included in the third learning input image, and a fourth having a second candidate scale. Of the building, the second candidate detector learned using the learning input image of the above and the teacher data of the information indicating the shapes of the plurality of buildings included in the fourth learning input image. Based on the evaluation unit that evaluates the shape detection accuracy and the detection accuracy evaluated by the evaluation unit, one of the first candidate detector and the second candidate detector is used in the first building. A building extraction system further comprising a detector and a detector selection unit for selection as any of the second building detectors.

(7) In any of (1) to (6), the integrated unit outputs the output of the first building detector and the output of the second building detector with respect to the feature information of the input input image. A building extraction system that determines an area recognized as a building in any of the above to be an area with a building.

It is a figure which shows an example of the hardware composition of the building extraction system which concerns on embodiment of this invention. It is a block diagram which shows the functional structure of a building extraction system. It is a figure explaining the kind of a learning detector. It is a figure explaining the difference of scale. It is a figure which shows the outline of the structure of the learning detector. It is a figure explaining the layer contained in the learning detector of a pooling model. It is a figure explaining the layer included in the learning detector of the dilation model. It is a figure explaining an example of the layer structure in the extended convolution operation. It is a flow diagram which shows an example of the process which makes a learning detector learn. It is a flow diagram which shows an example of the processing of the learning execution part for each of window images. It is a figure which shows an example of a teacher data. It is a flow diagram which shows an example of the process which evaluates a learning detector. It is a figure which shows the evaluation result. It is a figure explaining the outline of the process of determining the area of a building. It is a flow chart which shows the flow of the process which generates the whole output image from the process target image.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. Among the components that appear, those having the same function are designated by the same reference numerals, and the description thereof will be omitted.

In the building extraction system according to the present embodiment, an aerial photograph, a satellite image, or the like (aerial photograph or satellite image) obtained by taking a ground surface as a target area of the processing for extracting a building is applied to a building detector which is a trained model using a neural network. It may be an ortho image based on a satellite image, and is referred to as a “processed image” below), and the building area is determined and extracted based on the image output from the building detector. The building extraction system uses three building detectors when identifying a building from the image to be processed. Further, each of the three building detectors is configured to detect buildings having an area in the range of S, M, and L with higher accuracy. For example, the area range S is less than 45 m ² , the area range M is 45 m ² or more and less than 131 m ² , and the area range L is 131 m ² or more. Generally, the buildings belonging to the area range L correspond to condominiums and large commercial facilities, the area range M corresponds to apartments and retail stores, and the area range S corresponds to general houses.

Further, in the building extraction system according to the present embodiment, there are a plurality of building detectors having different types of neural networks and scales (scales) of input learning images for each of the area ranges S, M, and L. The best building detector is selected from a plurality of building detectors for each of the area ranges S, M, and L, and the selected building detector is the building area from the data to be processed. Is used to detect.

FIG. 1 is a diagram showing a hardware configuration of a building extraction system according to an embodiment of the present invention. The building extraction system includes a learning server 1. The learning server 1 is a server computer, and includes a processor 11, a storage unit 12, a communication unit 13, and an input / output unit 14.

The processor 11 operates according to the program stored in the storage unit 12. Further, the processor 11 controls the communication unit 13 and controls the device connected to the input / output unit 14. Here, the processor 11 may include a so-called CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) used as a parallel computer. The above program may be provided via the Internet or the like, or may be stored and provided in a computer-readable storage medium such as a flash memory or a DVD-ROM. ..

The storage unit 12 is composed of a memory element such as a RAM or a flash memory and a hard disk drive. The storage unit 12 stores the above program. Further, the storage unit 12 stores information and calculation results input from each unit.

The communication unit 13 realizes a function of communicating with other devices, and is configured by, for example, an integrated circuit of a wired LAN. The communication unit 13 transmits / receives information to / from other devices based on the control of the processor 11. Further, the communication unit 13 inputs the received information to the processor 11 and the storage unit 12. The communication unit 13 is connected to another device by, for example, a LAN.

The input / output unit 14 includes a video controller that controls a display output device, a controller that acquires data from the input device, and the like. Input devices include keyboards, mice, touch panels, and the like. The input / output unit 14 outputs display data to the display output device based on the control of the processor 11, and acquires the data input by the user operating the input device. The display output device is, for example, a display device connected to the outside.

Next, the outline of the function of the building extraction system will be described. FIG. 2 is a block diagram showing a functional configuration of a building extraction system. The building extraction system functionally includes a learning data acquisition unit 51, a learning execution unit 52, a learning detector set 53, an evaluation data acquisition unit 56, an evaluation execution unit 57, a detector selection unit 58, and the like. It includes an execution detector set 61, a target data input unit 65, an output acquisition unit 66, a filter unit 67, an integration unit 68, and an image output unit 69. These functions are mainly realized by the processor 11 executing the program stored in the storage unit 12 and accessing the data stored in the storage unit 12. All of these functions may be executed by the learning server 1, or some of the functions may be executed by another server. For example, the functions of the target data input unit 65, the execution detector set 61, the output acquisition unit 66, the filter unit 67, the integrated unit 68, and the image output unit 69 are the processor 11, the storage unit 12, the communication unit 13, and the input / output unit 14. It may be realized by another server having.

The learning detector set 53 has a plurality of learning detectors 54. In the present embodiment, the number of learning detectors 54 is 6, and each of the learning detectors 54 has a common portion 540 in which common learning is performed regardless of the area ranges S, M, and L, and an area range of each. It has individual units 541, 542, 543 for learning according to S, M, and L. Each of the learning detectors 54 learns about different combinations of neural network types and input training image scales.

The learning data acquisition unit 51 acquires a learning input image and teacher data indicating the shape of the building included in the learning input image. The learning execution unit 52 trains the learning detector 54 using the input image for learning and the teacher data.

The evaluation data acquisition unit 56 acquires an evaluation input image and correct answer data indicating the shape of the building included in the evaluation input image. The evaluation input image and the correct answer data may be the same as the learning input image and the teacher data. The evaluation execution unit 57 evaluates the detection accuracy of the shape of the building for each of the individual units 541, 542, 543 for each of the learning detectors 54 using the evaluation input image and the correct answer data.

The detector selection unit 58 selects the learning detector 54 that detects the building with respect to the input target data for each of the area ranges S, M, and L, based on the detection accuracy evaluated by the evaluation execution unit 57. At least a part of the selected learning detector 54 is used as the execution detectors 62, 63, 64 constituting the execution detector set 61. More specifically, the combination of the common part 540 and the individual part 541 included in the learning detector 54 selected for the area range S is included in the execution detector 62 corresponding to the area range S, the common part 620. And the individual part 621. The combination of the common part 540 and the individual part 542 included in the learning detector 54 selected for the area range M becomes the common part 630 and the individual part 631 included in the execution detector 63 corresponding to the area range M. .. The combination of the common part 540 and the individual part 543 included in the learning detector 54 selected for the area range L becomes the common part 640 and the individual part 641 included in the execution detector 64 corresponding to the area range L. ..

The target data input unit 65 acquires an input target image, processes the input target image as necessary, and inputs the input target image to the execution detectors 62, 63, 64. The output acquisition unit 66 acquires the output image output by the execution detectors 62, 63, 64.

The filter unit 67 removes the buildings included in the output images of the execution detectors 62, 63, 64 based on the area, and generates a filtered output image.

The integration unit 68 integrates the filtered output images of the execution detectors 62, 63, 64. The integration unit 68 generates an image in which the area recognized as a building in any of the output images of the execution detectors 62, 63, and 64 is determined to be a certain area of the building.

The image output unit 69 outputs the image integrated by the integration unit 68 to the storage unit 12 or the display output device.

Next, the details of the learning detector set 53 and the learning detector 54 included therein will be described. FIG. 3 is a diagram illustrating the types of the learning detector 54. “No” in the table shown in FIG. 3 indicates a number assigned to the six learning detectors 54. "Scale" indicates the scale of the learning input image input to the learning detector 54 of that number, and the learning input image prepared at the beginning is adjusted (enlarged or reduced as necessary) at the magnification indicated on the scale. , A learning input image cut out so as to have the same number of pixels regardless of the scale (hereinafter, the cut out learning input image is referred to as a “window image”) is input to the learning detector 54. The "model type" indicates the type of the neural network constituting the inside of the learning detector 54 of the number. "Polling" indicates that it is a model that combines a convolutional layer and a pooling layer (hereinafter referred to as "pooling model") in CNN (Convolutional Neural Network), and "Dilation" is a convolutional layer that performs an extended convolutional operation. It is shown that it is a model using (hereinafter referred to as "dilation model").

FIG. 4 is a diagram illustrating the difference in scale. FIG. 4A is an example of a window image when the scale is 0.5 times, and FIGS. 4B and 4C are window images when the scale is 1x and 2x, respectively. This is an example. The window images shown in FIGS. 4A to 4C include the same area. The number of pixels of the window image is Px × Py. The values of Px and Py may be, for example, 32 or 64. The learning input image when the scale is 0.5 times is obtained by reducing (thinning) the learning input image when the scale is 1.0 so that the number of vertical and horizontal dots is halved. The learning input image when the scale is 2.0 times is enlarged so that the number of vertical and horizontal dots is doubled (linear interpolation between dots, etc.). It is obtained by arranging dots by). Enlarging or reducing the learning input image is performed by the learning data acquisition unit 51.

FIG. 5 is a diagram showing an outline of the configuration of the learning detector 54. As described above, the learning detector 54 has a common unit 540 and individual units 541, 542, 543. The common part 540 has a plurality of layers, and the individual parts 541, 542, 543 have the same number of layers. The adjusted input image for learning is input to the first layer of the common part 540, and the feature information which is the output of the last layer is input to the first layer of each of the individual parts 541, 542, 543. .. The output of the individual portions 541, 542, 543 is, for example, an image of 16 × 16 dots, and each dot indicates the existence probability of the building at the position of the dot.

FIG. 6 is a diagram illustrating layers included in the learning detector 54 of the pooling model, and FIG. 6 shows each layer in the order of processing. In the column of affiliation, the layer described as "common" exists in the common unit 540, and the layer described as "individual" exists in the individual units 541, 542, 543. Here, the layers described in "individual" exist in each of the individual portions 541, 542, 543. The processing type indicates the type of each layer, "input" indicates an input layer, "convolution" indicates a convolution layer, and "pooling (s2)" indicates a pooling layer having a stride (kernel application interval) of 2. There is. The kernel size is a parameter that represents the size of the convolution filter. Here, corresponding to the processing target being an image, the kernel is two-dimensional, and the kernel size value "k" means that it is a "k × k" filter. The "number of feature maps" of each layer is the number of feature maps extracted in the layer, and is also called a channel. The stride is 1 unless otherwise specified, and the description for each layer is omitted.

FIG. 7 is a diagram illustrating layers included in the learning detector of the dilation model. The description of FIG. 7 is also based on the description of FIG. 6, but the layer of “convolution” in the dilation model indicates an extended convolution layer, and the setting of the expanded convolution layer is shown in the column of expansion coefficient.

The extended convolution operation will be further described. FIG. 8 is a diagram illustrating an example of a layer structure in the extended convolution operation. The input image such as the input image for learning is spatially two-dimensional data, but here, for the sake of simplification of illustration and explanation, the input data to the learning detector 54 will be simplified and described as one-dimensional data. Specifically, a plurality of "○" marks arranged in the horizontal direction in the input layer located at the bottom in FIG. 8 constitute input data. The element 30 of the input data represented by the “◯” mark corresponds to a pixel (or a pixel value) in the input image. The convolutional layer shown in FIG. 8 is a so-called feature extraction layer, and the description of the layer following the feature extraction layer is omitted.

The neural network shown in FIG. 8 has seven convolution layers as feature extraction layers, and each convolution layer performs an extended convolution operation. The convolution layer of the first layer located above the input layer performs an expansion convolution operation with an expansion coefficient d = 1. Specifically, a convolution operation is performed on each of the plurality of units 31 represented by "○" in the first layer, and each unit 31 multiplies the values of two adjacent elements 30 of the input layer and adds them together. Output the value.

The second convolutional layer performs an extended convolution operation with an expansion coefficient d = 2. Specifically, a convolution operation is performed on each of the plurality of units 32 represented by "○" in the second layer, and each unit 32 multiplies the output value of every other unit 31 in the first layer by a weight. The added value is output.

Further, the convolution layer of the third layer performs an expansion convolution operation having an expansion coefficient d = 3, and each unit 33 represented by the “○” mark of the third layer is the output value of every three units 32 in the second layer. The value obtained by multiplying the weights and adding them is output, the convolutional layer of the 4th layer performs the extended convolution operation with the expansion coefficient d = 4, and each unit 34 represented by the “○” mark of the 4th layer is the 3rd layer. The output value of every seven units 33 is multiplied by a weight and added together to output the value. Each unit 35 of the fifth layer performs an extended convolution operation of d = 3, and each unit 36 of the sixth layer and each unit 37 of the seventh layer perform an extended convolution operation of d = 2, d = 1, respectively. I do.

Here, in the structure of the feature extraction layer shown in FIG. 8, the portion consisting of the first layer to the fourth layer is referred to as a front end portion, and the portion consisting of the subsequent fifth layer to the seventh layer is referred to as a local feature extraction portion. I will call it. The front end portion is a plurality of convolution layers following the input layer, and in the front end portion, the expansion coefficient d increases to the maximum value in the feature extraction layer according to the order of the convolution layers. On the other hand, the local feature extraction unit is a plurality of convolutional layers following the front end portion, and the expansion coefficient of the local feature extraction unit decreases according to the order of the convolutional layers.

FIG. 8 illustrates the connection relationship between the units of the first layer to the sixth layer and the input layer convoluted to the output of one unit 37 of the seventh layer by a line. In the extended convolution operation, the scope of application of the kernel is expanded exponentially according to the expansion coefficient d. For example, the kernel of the convolution operation of d = 1 to 4 in FIG. 8 is a filter that convolves two inputs, that is, a filter of size 2, but one of the two inputs convolved by the kernel of d = 1. The spacing in the sequence of dimensional data is 1, whereas the spacing between the two inputs convolved by the kernel with d = 2 is 2, and at d = 3, the spacing is 4, and at d = 4, the spacing is 4. The interval is 8. That is, the interval is set to 2 ^d-1 .

As can be seen from the connection relationship between the unit and the input layer in the front end part, in the extended convolution operation, the receptive field can be expanded with a small number of layers by expanding the applicable range of the kernel. Since the receptive field is expanded only by convolution, the pooling layer used in a general CNN becomes unnecessary, and the resolution deterioration due to the pooling layer can be avoided. Further, while expanding the applicable range, the increase of the weight parameter is suppressed by thinning out the elements within the range and convolving only a part of the remaining elements.

On the other hand, a structure in which layers are stacked so that the expansion coefficient d increases in order, such as the front end part, has a problem that the correlation between neighboring units in the uppermost layer is weakened and a problem that it is difficult to pick up local characteristics of input data. Have. The local feature extraction unit is provided to solve this problem, and by combining the front end unit and the local feature extraction unit, the problem that the correlation between neighboring units is weakened in a unit with the 7th layer, and the second The problem that the local information that the units 31a and 31b of one layer are adjacent to each other cannot be grasped is solved.

In other words, by providing a local feature extraction section after the front end section, the front end section actively uses the extended convolution operation to obtain context without reducing the resolution at all, and the local feature extraction section. Now, the local features distributed by the front end part are aggregated. As a result, contextual information and local feature information can be effectively utilized, and even small and dense objects can be recognized.

Next, the details of the process of training the learning detector 54 described so far by using the learning input image according to the scale and the teacher data indicating the shape of the building included in the learning image will be described. ..

FIG. 9 is a flow chart showing an example of a process for learning the learning detector 54. FIG. 9 shows the processing of the learning data acquisition unit 51 and the learning execution unit 52, and the learning detector 54 is learned by this processing. Further, the process shown in FIG. 9 is performed for each learning detector 54 by the number of repetitions.

The learning data acquisition unit 51 acquires a learning image stored in the storage unit 12 (step S101). The learning image is an aerial photograph, a satellite image, or the like (which may be an ortho image based on the aerial photograph or the satellite image) taken on the ground surface as the target area of the processing for extracting the building. Next, the learning data acquisition unit 51 sets the size of the learning image to match the scale of the learning detector 54 (step S102). For example, if the scale of the learning detector 54 is 0.5 times, the learning image is reduced to 0.5 times, and if the scale is 2 times, the learning image is enlarged 2 times. Even if a plurality of learning images corresponding to each of the scale types are prepared in advance instead of the processing of step S102, and the learning data acquisition unit 51 reads the images corresponding to the scale of the learning detector 54. good.

Then, the learning execution unit 52 cuts out a window image to be input to the learning detector 54 from the learning image set to match the scale (step S103). The window image has a size of Px × Py, and a position is randomly selected from one learning image, and the window image is cut out from the learning image based on the selected position.

The learning execution unit 52 inputs a window image cut out from the learning image, and trains the learning detector 54 by comparing the output with the teacher data (step S104).

FIG. 10 is a flow chart showing an example of the processing of the learning execution unit 52 for each of the window images, and is a diagram for explaining the processing in step S104 in more detail. In step S104, first, the learning execution unit 52 inputs the window image cut out from the learning image to the common unit 540 of the learning detector 54 (step S121). As a result, the common unit 540 of the learning detector 54 processes the window image, and the individual units 541, 542, 543 further process the output of the common unit 540. Then, the learning execution unit 52 acquires the output images of the individual units 541, 542, 543 of the learning detector 54 (step S122). Here, in the following, the output image of the individual unit 541 corresponding to the area range S is the output image (S), the output image of the individual unit 542 corresponding to the area range M is the output image (M), and the area range L. The output image of the individual unit 543 corresponding to the above is referred to as an output image (L). Further, the output images of the individual portions 541, 542, 543 are collectively referred to as output images (S, M, L). Here, the value of each dot of the output image (S, M, L) indicates the existence probability of the area of the building.

Next, the learning execution unit 52 calculates an error between the output image (S, M, L) of the learning detector 54 and the teacher data (step S123). Here, the teacher data is information indicating the shape of the building included in the learning image data.

FIG. 11 is a diagram showing an example of teacher data. The teacher data shown in FIG. 11 is a bitmap image corresponding to the learning image including the window image shown in FIG. In the teacher data shown in FIG. 11, the area of the building whose area belongs to the range S (for example, A), the area of the building belonging to the range M (for example, B), and the area of the building belonging to the range L (for example, C) are included. It is distinguished. The teacher data may be, for example, an image in which the dot value of the area without a building is set to 0 and the dot value of the area of the building having the areas S, M, and L is set to 1, 2, and 3, respectively. .. Further, the teacher data includes an image in which the dot value of the area of the building whose area belongs to the range S is 1, an image in which the dot value of the area of the building whose area belongs to the range M is 1, and the area L. It may be an image corresponding to a plurality of layers with an image in which the dot value of the area of the building belonging to is 1.

In the calculation of the error, the learning execution unit 52 cuts out an image at a position corresponding to 16 × 16 dots in the center of the window image of the learning image from the teacher data, and with each of the output images (S, M, L). , Compare with teacher data. Here, the learning execution unit 52 calculates an error from the output image (S) for the area of the teacher data without a building and the area of the building belonging to the range S, but the area of the building belonging to the ranges M and L. Does not calculate the error. Similarly, the learning execution unit 52 does not calculate the error with the output image (M) for the area of the building belonging to the ranges S and L, and the error with the output image (L) with respect to the area of the building belonging to the ranges S and M. Is not calculated. As a result, learning proceeds so that each of the individual portions 541, 542, 543 is suitable for detecting a building in the area ranges S, M, and L.

Next, the learning execution unit 52 changes the values of parameters such as weights in the individual units 541, 542, 543 by an error back propagation method (backpropagation) or the like based on the calculated error (step S124). .. Further, the learning execution unit 52 integrates the error to be propagated from the highest layer of each of the individual units 541, 542, 543 to the lowest layer of the common unit (step S125), and based on the integrated error, the error is calculated. The value of the parameter such as the weight in the common portion 540 is changed by the back propagation method or the like (step S126).

The learning process shown in step S103 and step S104 (FIG. 9) is repeated until all the window images used for learning are acquired from a certain learning image. This set of processes is repeated for each of all the learning detectors 54, thereby learning each learning detector 54. Here, instead of the process of step S103, a process of collectively cutting out a plurality of window images used for learning may be performed. In this case, the process of step S104 may be repeatedly executed so that the process of inputting the window image and training the learning detector 54 is performed for each of the cut out window images.

Next, in the process of evaluating the learned learning detector 54 and selecting the learning detector 54 for actually executing the process of extracting the building area from the image to be processed as the execution detectors 62, 63, 64. The details will be described.

FIG. 12 is a flow chart showing an example of processing for evaluating the learning detector 54. In this process, first, the evaluation data acquisition unit 56 acquires an evaluation image and correct answer data from the storage unit 12 (step S201). The evaluation image may be the same as the learning image or may be different. The scale of the evaluation image is the same as that of the learning image. The correct answer data is an image showing the area of the building belonging to each of the area ranges S, M, and L in the evaluation image, and when the evaluation image and the learning image are the same, the correct answer data is the teacher data. good. Further, although not shown in FIG. 12, the evaluation data acquisition unit 56 sets the size of the evaluation image to match the scale of the learning detector 54, similarly to the learning data acquisition unit 51.

Next, the evaluation execution unit 57 cuts out a window image to be input to the learning detector 54 from the evaluation image (step S202). More specifically, the evaluation execution unit 57 cuts out a window image so that a predetermined number of dots are deviated from the window area cut out so far. The predetermined number of dots can be any size of 1 dot or more and 16 dots or less. The upper limit of 16 corresponding to the predetermined number corresponds to the output of the learning detector 54 being an image of 16 × 16 dots. The predetermined number is equal to or less than the vertical or horizontal size of the output of the learning detector 54. The evaluation execution unit 57 inputs the window image cut out from the evaluation image to the learning detector 54 (step S203), and outputs the respective output images (S, M, 543) of the individual units 541, 542, 543 of the learning detector 54. L) is acquired (step S204). Here, the evaluation execution unit 57 binarizes the acquired output image based on whether the value of the existence probability of each dot is larger or smaller than the threshold value, and stores the binarized output image in the storage unit 12. do. In the following processing, the output image refers to the binarized output image. Then, the processes of steps S202 to S204 are repeated until the process of the learning detector 54 is performed for all the window images (see step S205).

When the output images (S, M, L) for all the window images are obtained, the evaluation execution unit 57 together with the whole image (S) in which the output images (S) are arranged at the positions corresponding to those window images. , The whole image (M) in which the output image (M) is arranged at the position corresponding to those window images, and the whole image (L) in which the output image (L) is arranged at the position corresponding to those window images. , Are generated (step S206). More specifically, the evaluation execution unit 57 arranges the output images (S, M, L) so as to be offset by a predetermined number of dots from each other so as to correspond to the arrangement of the window images, so that the entire image (S, M, L) can be arranged. L) is generated. Here, when a predetermined number of dots, which is the size of the deviation when cutting out the window image, is smaller than 16 dots, at least some of the dots in the output image (S, M, L) obtained from each window image are It overlaps with the output images (S, M, L) for other window images. For dots whose positions overlap in the output of a plurality of window images, the evaluation execution unit 57 uses the average value obtained by averaging the dot values of the output image as the dot value in the entire image (S, M, L). As a result, even when the boundaries of adjacent output images (S, M, L) are not smoothly connected, it is possible to prevent the inconsistency caused by the inconsistency from appearing in the entire image.

Then, the evaluation execution unit 57 compares the entire image with the correct answer data, and evaluates the accuracy of each of the individual units 541, 542, 543 of the learning detector 54 (step S207). In the accuracy evaluation, for example, the evaluation execution unit 57 obtains the ratio (Recall) in which the area determined to be a building in the output image (S) exists in the area where the building belonging to the area range S exists in the correct answer data. Do it by. The evaluation execution unit 57 similarly evaluates the accuracy in the area of the building belonging to the area ranges M and L of the correct answer data and the area of the building existing in the output image (M) and the output image (L). ..

The accuracy of one learning detector 54 is evaluated by the processing of steps S202 to S207. Then, when the evaluation execution unit 57 has not evaluated the accuracy of all the learning detectors 54, the process from step S202 is repeated (step S208), whereby the evaluation execution unit 57 causes all the learning detectors 54. Evaluate the accuracy of.

FIG. 13 is a diagram showing the evaluation results by the evaluation execution unit 57. “No” in FIG. 13 indicates a number assigned to the learning detector 54, as shown in FIG. In the example of FIG. 13, for the output of the individual unit 541 whose area range is S, the learning detector 54 having a scale of 1.0 times and a dilation model has the highest accuracy. Regarding the output of the individual unit 542 having an area range of M, the output of the individual unit 543 having a scale of 1.0 times, the learning detector 54 of the pooling model having the highest accuracy, and the area range of L being L. The scale is 0.5 times and the learning detector 54 of the pooling model is the most accurate.

When the accuracy of the learning detector 54 is evaluated, the detector selection unit 58 sets the learning detector 54 with the highest accuracy as the execution detectors 62, 63, 64 for each of the area ranges S, M, and L. Select (step S209). The execution detector 62 is a combination of a common part 540 (hereinafter referred to as a common part 620) and an individual part 541 (hereinafter referred to as an individual part 621) included in the learning detector 54 having the highest accuracy in the area range S. be. The execution detector 63 is a combination of a common part 540 (hereinafter referred to as a common part 630) and an individual part 542 (hereinafter referred to as an individual part 631) included in the learning detector 54 having the highest accuracy for the area range M. be. The execution detector 64 is a combination of a common part 540 (hereinafter referred to as a common part 640) and an individual part 543 (hereinafter referred to as an individual part 641), which are originally included in the learning detector 54 having high accuracy in the area range L. Is.

Here, as can be seen from the description in FIG. 13, since the dilation model tends to catch small changes as compared with the pooling model, the dilation model is advantageous when the area range (maximum value) is small. Therefore, the pooling model is advantageous for those with a large area range. In addition, when the scale is small, detailed information is reduced, but it tends to be easier to determine the shape of a large-scale building. Therefore, if the area range (maximum value) is small, the larger scale is advantageous, and if the area range is large, the smaller scale is advantageous.

Therefore, also in the example of FIG. 13, as the execution detector 62 corresponding to the one having a small maximum value in the area range, the learning detector 54 having a large scale of 1.0 times and being a dilation model is selected. As the execution detector 64 corresponding to the one having a large maximum value in the area range, the learning detector 54, which has a small scale of 0.5 times and is a pooling model, is selected.

The detector selection unit 58 simply stores the information indicating the learning detector 54 to be the target for inputting the processing target image and acquiring the output image by the target data input unit 65, which will be described later, in the storage unit 12, so that the learning detector can be detected. 54 may be selected, or the learning detector 54 may be executed by copying the common part 540, the individual part 541, etc. of the selected learning detector 54 as the substance of the execution detectors 62, 63, 64. It may be selected as 62, 63, 64.

Next, the process of determining the area of the building from the image to be processed will be described using the execution detectors 62, 63, 64. FIG. 14 is a diagram illustrating an outline of a process for determining an area of a building.

First, the target data input unit 65 inputs the processing target image to the execution detector 62 suitable for the area range S, and the output acquisition unit 66 outputs the entire output image (S) based on the output of the execution detector 62. Acquire (step S301). The whole output image (S) is an image showing an area where a building is determined to exist by the execution detector 62 for the whole image to be processed. Similarly, the overall output image (M) and the overall output image (L), which will be described later, are images showing areas where it is determined by the execution detectors 63 and 64 that a building exists, respectively.

FIG. 15 is a flow chart showing a flow of processing for generating an overall output image from a processing target image, and is a diagram for explaining the processing in step S301 in detail. First, the target data input unit 65 adjusts the scale of the image to be processed to the scale set in the execution detector 62 (step S321). When the scale of the processing target image and the scale of the execution detector 62 are different, the target data input unit 65 adjusts the scale by enlarging or reducing the processing target image. Next, the target data input unit 65 cuts out a window image from the processed target image to which the scale has been adjusted (step S322). Since the method of cutting out the window image from the size of the window image and the image to be processed is the same as the method of cutting out the window image from the evaluation image, the description thereof will be omitted. Next, the target data input unit 65 inputs a window image to the execution detector 62 (step S323). Then, the execution detector 62 performs a process of detecting the area of the building with respect to the input window image, and the output acquisition unit 66 acquires the output image of the execution detector 62 (step S324). Here, although not shown, the output acquisition unit 66 binarizes the acquired output image based on whether the value of the existence probability of each dot is larger or smaller than the threshold value, and the binarized output image. Is stored in the storage unit 12. In the following processing, the output image refers to the binarized output image. Then, the processes of steps S322 to S324 are repeated until the process of the learning detector 54 is performed for all the window images (see step S325).

The building detector corresponds to the individual parts 621, 631, 641 of the execution detectors 62, 63, 64, and the feature information of the image to be processed input to the building detector is the common parts 620, 630, 640, respectively. It may be an output. The learning detector 54 and the execution detectors 62, 63, 64 do not have to include the common portions 540, 620, 630, and 640. In this case, the learning input image and the processing target image are input for each of the area ranges S, M, and L, and the feature information of the processing target image input to the building detector is simply the processing target image or its window image. It may be there.

When the output images for all the window images are obtained, the evaluation execution unit 57 generates an overall output image (S) in which the output images are arranged at positions corresponding to those window images (step S326).

Next, the filter unit 67 applies an area-based filter to the entire output image (S) (step S302). More specifically, in this process, the filter unit 67 calculates the area (obtained from the number of dots and the scale of the area) of the area where the building is determined to exist in the overall output image (S), and the area is calculated. The area that is not within the permissible range according to the area range S is deleted from the overall output image (S). Specifically, the permissible range is less than 89.2 m ² . The processing of the filter unit 67 may not be performed.

Further, the target data input unit 65 inputs the processing target image to the execution detector 63 suitable for the area range M, and the output acquisition unit 66 outputs the entire output image (M) based on the output of the execution detector 63. Acquire (step S303). Since the details of this process are the same as the process of acquiring the entire output image (S) from the execution detector 62, detailed description thereof will be omitted.

Next, the filter unit 67 applies an area-based filter to the entire output image (M) (step S304). More specifically, in this process, the filter unit 67 calculates the area of the area where the building is determined to exist in the overall output image (M) (obtained from the number of dots and the scale of the area), and the area is calculated. Areas that are not within the permissible range according to the area range M are deleted from the overall output image (M). Specifically, the permissible range is 22.3 m ² or more and less than 89.2 m ² .

Further, the target data input unit 65 inputs the processing target image to the execution detector 64 suitable for the area range L, and the output acquisition unit 66 outputs the entire output image (L) based on the output of the execution detector 64. Acquire (step S305). Since the details of this process are the same as the process of acquiring the entire output image (L) from the execution detector 62, detailed description thereof will be omitted.

Next, the filter unit 67 applies an area-based filter to the entire output image (L) (step S306). More specifically, in this process, the filter unit 67 calculates the area of the area where the building is determined to exist in the overall output image (L) (obtained from the number of dots and the scale of the area), and the area is calculated. The area that is not within the permissible range according to the area range L is deleted from the overall output image (M). Specifically, the permissible range is 65.4 m ² or more.

Then, the integration unit 68 executes a process of enlarging or reducing at least one of these so that the scales of the overall output image (S), the overall output image (M), and the overall output image (L) match. (Step S307). This process may be performed before the process of the filter unit 67.

The integration unit 68 integrates the processed overall output image (S), overall output image (M), and overall output image (L) (step S308). In other words, the integration unit 68 determines that the area recognized as a building in any one of the total output image (S), the total output image (M), and the total output image (L) is a region with a building, and the determination is made. Generates an integrated image showing the area that has been removed. More specifically, the integration unit 68 creates an integrated image by taking the logical sum of the dots of the filtered overall output image (S), overall output image (M), and overall output image (L). Generate. Here, it is assumed that the dots of the overall output image (S), the overall output image (M), and the overall output image (L) are 1 in the area where it is determined that the building exists, and 0 in the area where the building is not present. ..

Then, the image output unit 69 outputs the image generated by the integration unit 68 to the storage unit 12 or the display output device.

Images in which the area of the building was determined were acquired using execution detectors 62, 63, 64 having scales and model types suitable for each of the area ranges S, M, and L, and further, the integrated unit 68 used them. By integrating the images, it is possible to improve the accuracy of the building determined from the processed image and reduce oversight in particular.

For example, based on the evaluation result shown in FIG. 13, the detector selection unit 58 has a scale of 1.0 times and a dilation model and a scale of 1.0 times as execution detectors 62, 63, 64, respectively. When the pooling model, the learning detector 54 of the pooling model with a scale of 0.5 times is selected, in one experiment, the value of Recall, which is an index of oversight, is 87.0%, and the execution detectors 62, 63, 64. As for the 82.0%, which is the value when the scale is 1.0 times and the pooling model, and the execution detectors 62, 63, 64, the scale is 1.0 times and the dilation model is used. It exceeds the value of 83.8% in the case. Here, the value of Recall is a number obtained by dividing the number of areas where it is determined that a building exists among the areas of the building given as the correct answer by the number of areas of the building given as the correct answer. This effect is very large because it is generally not easy to reduce oversights in determining the area of a building.

By using the building extraction system that combines the execution detectors 62, 63, and 64 described so far, structures and buildings of various sizes can be obtained with higher accuracy from remote sensing images such as aerial photographs and satellite images. You will be able to recognize it. Then, the building extraction system can be used to grasp new construction or loss of a building, and it is possible to obtain basic information on statistics on house changes. Furthermore, by being able to extract the building area with high accuracy, it is easier to grasp the temporal transition of each building, and the detailed attributes of the building (for example, detached house, etc.) from the size and shape of the extracted building area. It will also be easier to identify the type of building (type of building such as condominium or factory).

By automating the work of extracting information about the building from the image, it is possible to perform the work on a wide range of ground surfaces at low cost and at high speed.

Although the embodiments of the present invention have been described so far, various modifications can be made within the scope of the gist of the present invention. For example, the area range may be two or four or more instead of three. Also, the number of model types and the number of scale types may be different. Further, the individual part does not have to be optimized according to the range of the area of the building. Individual parts may be optimized according to groups classified by other methods, for example, the height of a building.

1 learning server, 11 processor, 12 storage unit, 13 communication unit, 14 input / output unit, 30 elements, 31,32,33,34,35,36,37 units, 51 learning data acquisition unit, 52 learning execution unit, 53 Learning detector set, 54 learning detector, 540 common part, 541,542,543 individual part, 56 evaluation data acquisition part, 57 evaluation execution part, 58 detector selection part, 61 execution detector set, 62,63,64 Execution detector, 620, 630, 640 common part, 621, 631, 641 individual part, 65 target data input part, 66 output acquisition part, 67 filter part, 68 integration part, 69 image output part.

Claims (5)

A teacher of information showing the shapes of a first learning input image having a first scale and the plurality of buildings included in the first learning input image for a plurality of buildings whose areas belong to the first range. The first building detector learned using the data,
For a plurality of buildings belonging to a second range whose area is different from the first range, a second learning input image having a second scale and a plurality of buildings included in the second learning input image. A second building detector trained using teacher data containing shape information,
The learning target area on the ground surface is photographed from the sky, and the feature information of the first input image having the first scale is input to the first building detector, and the first input image is the first scale. And an input unit that inputs the feature information of the second input image enlarged or reduced according to the ratio to the second scale to the second building detector.
An integrated unit that integrates the output of the first building detector with respect to the feature information of the first input image and the output of the second building detector with respect to the feature information of the second input image.
Including
The maximum value of the first range is larger than the maximum value of the second range.
The minimum value of the first range is larger than the minimum value of the second range.
The first scale is smaller than the second scale.
Building extraction system. A teacher of information showing the shapes of a first learning input image having a first scale and the plurality of buildings included in the first learning input image for a plurality of buildings whose areas belong to the first range. The first building detector learned using the data,
For a plurality of buildings belonging to a second range whose area is different from the first range, a second learning input image having a second scale and a plurality of buildings included in the second learning input image. A second building detector trained using teacher data containing shape information,
The learning target area on the ground surface is photographed from the sky, and the feature information of the first input image having the first scale is input to the first building detector, and the first input image is the first scale. And an input unit that inputs the feature information of the second input image enlarged or reduced according to the ratio to the second scale to the second building detector.
An integrated unit that integrates the output of the first building detector with respect to the feature information of the first input image and the output of the second building detector with respect to the feature information of the second input image.
Including
The maximum value of the first range is larger than the maximum value of the second range.
The minimum value of the first range is larger than the minimum value of the second range.
A first candidate detector and a second candidate detector provided for each of the first range and the plurality of ranges including the second range.
For each of the plurality of ranges, a third learning input image having a first candidate scale and information teacher data indicating the shapes of a plurality of buildings belonging to the range included in the third learning input image are teacher data. Included in the first candidate detector learned using the above, a fourth learning input image having a second candidate scale different from the first candidate scale , and the fourth learning input image. An evaluation unit that evaluates the detection accuracy of the shape of each of the second candidate detectors learned by using the teacher data of the information indicating the shapes of a plurality of buildings belonging to the range .
Based on the detection accuracy evaluated by the evaluation unit, one of the first candidate detector and the second candidate detector provided for the first range is referred to as the first building detector . Then, one of the first candidate detector and the second candidate detector provided for the second range is selected as the second building detector, and the first candidate is selected. Of the scales and the second candidate scales, the one corresponding to the selected one of the first candidate detector and the second candidate detector provided for the first range is the first scale. Of the first candidate scale and the second candidate scale, the one selected from the first candidate detector and the second candidate detector provided for the second range. Further includes a detector selection unit that selects the corresponding one as the second scale .
Building extraction system. In the building extraction system according to claim 1 or 2.
Further comprising a filter for removing the building included in the output of the first building detector and the building included in the output of the second building detector based on the area.
Building extraction system. In the building extraction system according to any one of claims 1 to 3.
In the integrated unit, the scales of the output of the first building detector with respect to the feature information of the first input image and the output of the second building detector with respect to the feature information of the second input image match. As described above, the process of enlarging or reducing at least one of the two outputs is executed, and the two outputs for which the process is executed are superimposed.
Building extraction system. In the building extraction system according to any one of claims 1 to 4 .
The integrated unit sets a region recognized as a building in either the output of the first building detector or the output of the second building detector with respect to the feature information of the input input image of the building. Judge as a certain area,
Building extraction system.

Images