JP2018097506A - Satellite image processing system and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To appropriately divide a satellite image in accordance with contents of the image in order to appropriately perform change extraction of the satellite image.SOLUTION: A satellite image processing system includes: a difference image generation part; and an image division part for acquiring an image to be compared to acquire difference images generated by the difference image generation part. The image division part includes: first block image group generation processing parts (S3a, S3b) for dividing a satellite image to acquire a first block image group of a prescribed size; classfication processing parts (S4a, S4b) for classifying each of first block images; and second block image group generation processing parts (S5a, S5b) for acquiring a second block image group by dividing the first block images on the basis of a class of each of the first block images or connecting the first block image to an adjacent first block image. The difference image generation part generates difference images of the plurality of satellite images to an area corresponding to second block images.SELECTED DRAWING: Figure 1A

Description

  The present invention relates to a satellite image processing system and method.

  The change extraction technique from the satellite image makes it possible to observe a change in the ground coverage in the same region, or to observe the appearance of a new feature that has not been heretofore over a wide area. Image processing is indispensable to obtain meaningful information from a huge amount of satellite images.

  As a pre-processing of the satellite image, a method of dividing into approximate regions by performing clustering has been studied. Japanese Patent Application Laid-Open No. 2004-228561 discloses a technique of dividing a satellite image into blocks, classifying the images according to texture, and dividing the image by clustering in which adjacent blocks and blocks having similar classes are connected.

JP 2005-196702 A

  The inventors examined the application of deep learning to improve the accuracy of extracting changes from multiple satellite images. In the field of image recognition, deep learning using supervised learning, particularly image recognition using a convolutional neural network (CNN), has achieved great results. An image recognition method using deep learning is effective when you want to extract a specific target from satellite images with high precision or when you do not want to recognize changes in brightness due to seasonal changes in the same feature as changes. it is conceivable that.

  In CNN, in order to recognize a target object from an image divided into blocks, it is necessary to first divide a satellite image into an appropriate block image. Since the satellite image is a very large image, it is inefficient to recognize the entire image comprehensively by dividing the block into CNNs. Further, when the size of the target object is not known, it is necessary to repeat the search for each block whose size has been changed, and the efficiency further deteriorates.

  For this reason, applying deep learning to extract change extraction from a satellite image is based on the premise that a block image that is likely to change is first extracted appropriately. When covering classification or object recognition is performed from a satellite image, it is effective to divide into regions as in Patent Document 1. However, the area divided in this way is not necessarily a block suitable for image comparison (difference extraction). This is because, for example, it is difficult to match images of blocks having a complicated texture and large, and there is a high possibility that a block having a uniform texture is erroneously extracted as a small change. Therefore, in order to appropriately extract changes from the satellite image, it is necessary to appropriately divide the satellite image according to the contents of the image.

  A processor, a memory, and a change extraction program that is read into the memory and executed by the processor. The change extraction program is an image to be compared to obtain a difference image generated by the difference image generation unit and the difference image generation unit. An image dividing unit for obtaining a first block image group generation processing unit for dividing a satellite image of a first resolution to obtain a first block image group of a predetermined size, and a first block image A classification processing unit that classifies each of the first block images belonging to the group based on one or more of texture, vegetation index, and elevation, and the first block image based on the class of the first block image Second block image group generation processing for obtaining a second block image group by dividing or combining with a first block image adjacent to the first block image Has the door, in the differential image generating unit generates a difference image of a plurality of satellite images against the corresponding region in a second block image generated by the second block image group generation processing unit.

  Image recognition accuracy can be improved by appropriately dividing the satellite image according to the content of the image and generating a difference image.

It is a figure which shows the processing flow of a present Example. It is a figure which shows the processing flow of a present Example. This is a hardware configuration of the present embodiment. Data and programs stored in the storage device 412. It is an example of the satellite image processed in a present Example. It is an example of a class definition table stored in the class database 203. It is a figure which shows an automatic learning process flow.

  A procedure for extracting changes by applying deep learning to a plurality of satellite images with different imaging conditions observed at the same point will be described. The imaging conditions include the satellite imaging angle, imaging timing, sensor performance, resolution, and the like. For example, if the shooting angle is different, even the same feature is shot in different ways, and if the shooting time is different, a difference in brightness between satellite images appears.

  FIG. 2 shows a hardware configuration of a system for realizing the present embodiment. A program for extracting changes in the satellite image of this embodiment, image data such as a satellite image, and other data necessary for image processing are stored in the storage device 412. Through the user interface 416 typified by a keyboard and a mouse, the system of this embodiment receives a user instruction. In response to a user instruction, the processor 413 reads a necessary program and image data from the storage device 412 into the memory 415 and executes the program. Since the satellite image is usually large-scale image data, when high-speed image processing calculation is required, the image data is transferred to the memory of the graphic processor 414 and its calculation support is used. The display device 411 is a display that displays images before and after image processing according to the present embodiment. These are connected through a bus 417.

  Note that FIG. 2 only shows a typical hardware configuration, and it goes without saying that various modifications are possible. For example, programs and image data may be stored in a plurality of storage devices. Alternatively, a network interface may be provided and connected to a storage device outside the system, and programs and image data may be stored in the storage device outside the system. Similarly, a user may be able to access the system from an external terminal via a network.

  FIG. 3 shows data and programs stored in the storage device 412. The storage device 412 includes, as data, a satellite image database (DB) 201 that stores satellite images, a DEM database 202 that stores DEM (Digital Elevation Model) data indicating the altitude of the ground surface, and class classification to be described later. A class database 203 for storing a reference, a teacher data database 204 for storing teacher data for extracting a difference image by applying deep learning, and a learning model database 205 for storing a connection state of a recognizer of a neural network are stored. Yes.

  The storage device 412 stores a change extraction program 101 and an automatic learning program 102 as programs. The change extraction program 101 includes an image dividing unit 111, a difference image generating unit 112, a change extracting unit 113, and a report generating unit 114. The automatic learning program 102 includes a determination receiving unit 121, a database updating unit 122, and a learning model updating unit 123. Each function of these programs will be described later.

  Hereinafter, the processing flow (FIGS. 1A and 1B) of the present embodiment will be described using the example of the satellite image shown in FIG. The rectangular satellite image 300 holds latitude / longitude information of vertices 301 and 302 located diagonally as coordinate information. With this coordinate information, the latitude / longitude of an arbitrary position of the satellite image can be specified, and alignment of different satellite images becomes possible. In the case of the satellite image 300, the left side is the land area 304 and the right side is the sea area 305 with respect to the boundary 303. On the land area 304, artificial objects such as a building 306, natural objects such as rivers 307, grasslands 308, and trees 309 are included as images. Note that the satellite image 300 in FIG. 4 is shown in a state where a part of the land area 304 is covered with a mask 311 indicated by hatching. This will be described later.

  The processing flow of FIGS. 1A and 1B shows a second satellite image A (comparison target image) such as the satellite image 300 shown in FIG. 4 and a second image captured under imaging conditions different from the first satellite image. Are compared with the satellite image B (comparison source image), and a difference image is extracted, and image recognition is performed by applying deep learning to the difference image.

  First, since a common area of a plurality of satellite images is a target, an image of a common area of the comparison target image and the comparison source image is cut out using coordinate information included in the satellite image. Considering the handling by a computer, it is desirable to cut out the common area by a rectangle. In order to perform image recognition from the difference images of a plurality of satellite images, a high-resolution satellite image is necessary. However, at the stage where a difference is extracted, a high resolution up to that point is not necessary. Therefore, in order to shorten the calculation time, it is desirable to prepare a satellite image with a reduced resolution in advance (S1a, S1b). For example, a low-resolution image is generated by using another thread of the processor 413 or parallel operation using the graphic processor 414 when reading the satellite image. In this way, the high-resolution satellite image 11a (11b) may be processed as it is to create the low-resolution satellite image 12a (12b), or at a time when it can be considered that the comparison target image and the comparison source image were taken at the same time. You may make it refer to the image | photographed low-resolution satellite image 12a '(12b').

  A common area is extracted using the coordinate information of the low resolution satellite image 12a (12a ') and the coordinate information of the low resolution satellite image 12b (12b') (S2). For example, the following processing steps will be described on the assumption that an area corresponding to the satellite image 300 in FIG. 4 is obtained as the common area.

  The low resolution satellite image is divided into a first block image group of a predetermined size (S3a, S3b). In the first block image, the size is such that separation between the sea and land and separation between mountains and flat ground can be performed. In this case, it may be divided in a wide area. For example, if it is sufficient to distinguish between a mountain and a flat land using a DEM image, the coarse size of the DEM image issued by the Geospatial Information Authority of Japan is equivalent to 250 m, so it is sufficient to divide it in units of 250 m. . For areas where the resolution of the DEM is fine, the size can be subdivided in proportion, but there is a risk of over-division. When comparing images that are over-divided, that is, images that are too small in size, it is easy to adjust the image for alignment, but the area that appears in common in the two images to be compared is small, and it can be compared. May become smaller. On the other hand, when underdivided images, that is, images that are too large, are compared with each other, the correction directions for alignment cancel each other and the image adjustment amount for alignment tends to be small. is there.

  In FIG. 4, the first block image 312 is displayed as a dotted square. A first block image group (312a, b, c... Is obtained) by dividing the common area into a predetermined size. Furthermore, if the common area is divided into the first block images without overlapping each other, there is a possibility that an area that cannot be compared at the end of the image appears. For example, when there is a small island 310 at the end of the first block image 312a, the area occupied by both the first block image 312a and the first block image 312d is small, and the information may be lost. There is. For this reason, in order to interpolate between the first block images, a rectangular area image 313a having the same size as the first block image and offset in coordinates is added as the first block image. As another method, when dividing into first block images, adjacent first block images may have overlapping regions of a certain width.

  Furthermore, in order to speed up the processing, the first block image outside the region of interest 314 (ROI: Region of Interest) that is known in advance that the target that the user wants to detect does not exist in the common region is It can be removed from the process. An area where it is known in advance that there is no target to be detected by the user may be excluded by the mask 311. If the mask area is a vector image, it is determined by determining whether the coordinates of the four corners of the first block image are included inside the polygon shape of the mask. Alternatively, if the mask area is a raster image, the pixel data of the satellite image corresponding to the mask area is invalidated by raster multiplication, and the first block image in which all the pixel data is invalidated is excluded from the subsequent processing. Conversely, in the case of ROI designation, on the contrary, the first block image that does not include a pixel corresponding to the ROI region 314 (a dashed-dotted circle region) is excluded from the subsequent processing.

  The first block images obtained in this way are classified (S4a, S4b). The first block image is classified according to the class definition table 500 stored in the class database 203. In the class definition table 500, a texture 501 representing the uniformity / complexity of an image, a vegetation index (NVDI) 502 that is an index indicating the distribution status and activity of vegetation, altitude, land use data, etc. A plurality of classes 505 are defined based on the map topographic information 503 (an example in which elevation is used as an index in the figure), and different division sizes of the second block images are defined for each class.

  The aim of classification is to classify land and sea, or land that is close to land use. For example, the ocean can be estimated by texture calculation because the luminance distribution is small and uniform unless the land or ship is reflected. For example, a statistic such as an average, variance, or moment calculated from the luminance histogram of an image can be used as an index, and the range of the index for each class is expressed in order to represent uniformity or complexity for each class. Define it. The mountainous area has a large NVDI, and can be estimated by extracting a high altitude portion from the DEM data. Moreover, you may make it refer to land use data for the urban area where artifacts gather. For this reason, the division size of the second block image defined according to the class is defined according to map topography information such as texture uniformity, vegetation index, and altitude. In general, classes with higher elevation should be divided smaller than classes with lower elevation, classes with higher vegetation index should be divided smaller than classes with lower vegetation index, and classes with lower uniformity are more uniform Should be divided smaller than the higher class. The former is less likely to be over-divided even if it is divided smaller, and is effective in obtaining a subsequent difference image. The classification may be performed based on one or more of the exemplified indexes. For example, if the region of interest is limited to flat land or the sea, the altitude may not be used as an index for class classification. Conversely, a class can be defined using an index other than the example.

  Next, a second block image is generated based on the class of the first block image (S5a, S5b). The second block image is generated by combining adjacent first block images or dividing the first block image. As in class 1 of the class definition table 500, in the case of an image with extremely high uniformity of texture, that is, an image with high uniformity, the probability of erroneous extraction increases by extracting a slight difference in value as a change. Therefore, the first block 312b including only the sea area 305 is combined with the adjacent block 312a, for example. Since the first block 312a includes an image of the small island 310, it is not necessary to detect and extract a slight luminance change at sea. On the other hand, the first block image other than class 1 is divided by the division size 506 defined in the class definition table 500. For example, in a mountainous area, the satellite image can be easily aligned by reducing the second block image. For example, in FIG. 4, both the second block image 321 obtained by combining the first block images 312a and 312b and the second block image 322 obtained by dividing the first block 312e are included. Thereby, it is possible to suppress the influence of excessive division and underdivision that occur when two different images are compared.

  Using the obtained coordinate information of the second block image group, a corresponding image region is cut out from the high resolution image (S6a, S6b). The cut out image is referred to as a “high-resolution second block image”. The above processing is realized by executing the function of the image dividing unit 111 of the change extraction program 101. That is, each step of the processing flow described in the processing flow of FIG. 1A is realized as a program part of the change extraction program 101. The same applies to the following.

  FIG. 1B shows a processing flow after FIG. 1A. First, alignment between the high-resolution second block images having the same coordinate information is performed. If the satellite image is not completely orthorectified, it is not possible to perfectly align only with position information such as latitude and longitude. This is because, even at the same position, if the shooting angle is different, the way the image is taken is also different. Therefore, before obtaining the difference image, alignment by image processing is necessary. For this purpose, feature points are extracted for each of the high-resolution second block images (S7a, S7b). Using the extracted feature points, the high-resolution second block images are aligned by image matching (S8). The difference image 14 is produced | generated by taking the brightness | luminance difference for every pixel about the high-resolution 2nd block images which aligned.

  When generating a difference image of a satellite image, if the difference is taken as it is, it may be affected by the overall brightness difference between images due to differences in shooting conditions, so an image of a brightness gradient with surrounding pixels is generated. Then, it is good to extract the difference. In order to form the difference image into a shape that makes it easier to extract the region, a method of masking the region by automating threshold determination and binarizing can be considered. The binarization threshold automatic decision algorithm may be changed based on information classified into the first block image. The binarized image is subjected to expansion and contraction processing to remove noise. As the difference extraction algorithm information 507 in the class definition table 500, an algorithm and parameters for extracting a difference image for each class are defined, and a difference image is generated in accordance with this.

  The above processing is realized by executing the function of the difference image generation unit 112 of the change extraction program 101.

  Next, an area with a large luminance change is extracted from the difference image 14 between the high-resolution second block image A13a and the high-resolution second block image B13b, and the coordinate information is acquired using the periphery of the area as the third block image ( S10). For the pixel set having a large luminance change in the difference image 14, a rectangle including the pixel set is selected around the pixel set, and the third block images 15a and 15b are generated by cutting out the region from the original satellite image. . The size of the rectangle may be a circumscribed rectangle, or the size may be determined by narrowing the target to the target to be extracted. If the size of the pixel set is too small, it may be handled as a set with an adjacent pixel set.

  Image recognition using CNN is performed on the third block image (S12a, S12b). Specifically, it is confirmed whether or not a target object corresponding to the teacher data stored in the teacher data database 204 exists. If it is determined that the target object exists, the shape of the determined target object is set to the third block. While extracting from an image, the area | region obtained by the difference image is updated to the shape of a target object.

  Based on the result of the image recognition S12, it is determined whether or not the change that the user is searching for has occurred in the region (S13). Specifically, if it is determined from any one of the third block images that the target object exists, the target object appears between when the comparison source image is captured and when the comparison target image is captured, Alternatively, it can be determined that there has been a change of being lost. Alternatively, if the presence of the target object cannot be determined from any third block image, it can be determined that the change being searched for by the user has not occurred in the region.

  The above processing is realized by executing the function of the change extraction unit 113 of the change extraction program 101. The image recognition method assumes deep learning such as CNN, but other recognition methods such as pattern matching can be used for the third block image.

  The change extracted by the change extraction unit 113 is presented to the user by means such as displaying it on the display device 411 (S14). As a result, the user can determine whether or not the change extracted by the change extraction unit 113 is really a change that the user is searching for. The above processing is realized by executing the function of the report generation unit 114 of the change extraction program 101.

  When deep learning is used for image recognition in the change extraction unit 113, it can be expected that the extraction accuracy improves as teacher data increases. For this reason, the automatic learning program 102 is operated as a separate process from the change extraction program 101, and the teacher data database 204 and the learning model database 205 are updated. This processing flow is shown in FIG.

  When the change extracted by the change extraction unit 113 is displayed on the display device 411 or the like, the user determines whether or not the displayed change is really a search for change, and inputs the determination result through the user interface 416. To do. The feedback from the user is accepted (S61), and the teacher data database 204 is updated (S62). For example, when the change to be extracted is the target object X, and it is determined that the change of the target object X is correctly extracted, the extracted third block image is used as learning data of “image of target object X”. Is registered in the teacher data database 204. Conversely, if it is determined that a change in Y has been extracted for the target object X, the extracted third block image is stored in the teacher data database 204 as learning data for “an image that is not the target object X”. sign up. Furthermore, each time the teacher database is updated a predetermined number of times or more and a certain amount of new learning data is accumulated in the teacher data database 204, the network connection (learning model) of the recognizers stored in the learning model database 205 is updated. (S63).

  In addition, the feedback extraction from the user can be expected to improve the accuracy of change extraction by adjusting the method of cutting out the image and the generation method of the difference image 14. For example, the accuracy of change extraction can be improved by adjusting the division size of the second block image for each class stored in the class database 203 and the criteria for determining difference extraction. The determination criterion for difference extraction is an algorithm used for difference extraction, a binarization algorithm after difference extraction, or the like. For example, the setting value of the difference extraction algorithm information 507 in the class definition table 500 is changed according to the feedback result. For example, when the number of feedbacks reaches a certain value, the setting value of a class whose error probability exceeds a certain value is updated.

  For example, since each class is defined by indices such as texture, vegetation index (NVDI), and altitude (DEM), the spectrum pattern is relatively close to the class adjacent to the class. For this reason, if the error probability of the adjacent class is small, the set value of the class may be brought close to the set value of the adjacent class. When the accuracy of change extraction is not improved before and after the update to approach the setting of the adjacent class, the division size of the second block image is changed. For example, the division size is enlarged / reduced in a certain step. If there are many oversights (not detected but changed due to user feedback), the luminance dispersion in the image tends to be too large, so it is desirable to reduce the image size. On the other hand, if there are many false detections (detected but not changed by user feedback), the value of the luminance dispersion in the image may be too small, so it is desirable to increase the image size. Since the difference extraction algorithm basically needs to be adaptively changed according to the luminance distribution in the block image, confirm that the luminance distribution has changed when the image division size changes. It is better to reset it so that the algorithm is appropriate for the luminance distribution. Note that steps S61, S62, and S63 in FIG. 6 are realized by executing the functions of the determination receiving unit 121, the database updating unit 122, and the learning model updating unit 123 of the automatic learning program 102, respectively.

  Although the embodiments have been described in detail above, various modifications are possible. For example, in a situation where a large number of satellite images can be prepared, a background image excluding a short-term change region can be generated by using an image obtained by averaging the pixels of the satellite image captured at the same point. By comparing the background image as a comparison source image with the comparison target image, the change (appearance or disappearance) of the target object can be extracted more easily. In this case, when extracting the difference image 14, pixel alignment should have been performed, so it is desirable to update the background image based on the second block image that has been aligned.

101: change extraction program, 102: automatic learning program, 111: image division unit, 112: difference image generation unit, 113: change extraction unit, 114: report generation unit, 121: determination reception unit, 122: database update unit, 123 : Learning model update unit, 201: satellite image database, 202: DEM database, 203: class database, 204: teacher data database, 205: learning model database, 411: display device, 412: storage device, 413: processor, 414: Graphic processor, 415: memory, 416: user interface, 417: bus.

Claims (15)

A processor;
Memory,
A change extraction program read into the memory and executed by the processor;
The change extraction program includes a difference image generation unit and an image division unit for obtaining an image to be compared in order to obtain a difference image generated by the difference image generation unit,
The image dividing unit includes a first block image group generation processing unit that divides a satellite image having a first resolution and obtains a first block image group having a predetermined size;
A classification processing unit that classifies each of the first block images belonging to the first block image group based on one or more of texture, vegetation index, and elevation;
Generation of a second block image group that obtains a second block image group by dividing the first block image based on the class of the first block image or combining the first block image with the first block image adjacent to the first block image A processing unit,
A satellite image processing system for generating a difference image of a plurality of satellite images for a region corresponding to the second block image generated by the second block image group generation processing unit in the difference image generation unit. In claim 1,
Among the first block images belonging to the first block image group, a satellite image that is a second block image by combining a first block image having a texture uniformity equal to or greater than a predetermined value with an adjacent first block image. Processing system. In claim 1,
In the case where the first block image of the first class is divided into a first size and the first block image of the second class is divided into a second size smaller than the first size to obtain a second block image. ,
The satellite image processing system, wherein the altitude defined as the second class is higher than the altitude defined as the first class. In claim 1,
In the case where the first block image of the first class is divided into a first size and the first block image of the second class is divided into a second size smaller than the first size to obtain a second block image. ,
The satellite image processing system, wherein the vegetation index defined as the second class is higher than the vegetation index defined as the first class. In claim 1,
In the case where the first block image of the first class is divided into a first size and the first block image of the second class is divided into a second size smaller than the first size to obtain a second block image. ,
The texture defined as the second class has a lower uniformity than the texture defined as the first class. In claim 1,
The change extraction program further includes a change extraction unit,
The said change extraction part is a satellite image processing system which performs image recognition which applied deep learning to the 3rd block image cut out from the satellite image of the 2nd resolution higher than the said 1st resolution based on the said difference image. In claim 6,
A teacher data database for storing teacher data used for the image recognition;
An automatic learning program that is read into the memory and executed by the processor;
The said automatic learning program is a satellite image processing system which has a database update part which receives the feedback from the user about the result of the said image recognition, and updates the teacher data stored in the said teacher data database. In claim 7,
A learning model database for storing a learning model of a recognizer used for the image recognition;
The said automatic learning program is a satellite image processing system which has a learning model update part which updates the said learning model at the timing when the said teacher data were updated more than predetermined times. In claim 7,
For each class of the first block image, a division size for dividing the first block image to obtain the second block image group, and difference extraction applied when the difference image generation unit generates the difference image. A class database storing a class definition table having algorithm information;
The satellite image processing system, wherein the automatic learning program includes a database update unit that receives feedback from a user regarding the result of the image recognition and updates the class definition table stored in the class database. Dividing a first satellite image of a first resolution to generate a first block image group of the first satellite image of a predetermined size;
Dividing the second satellite image of the first resolution to generate a first block image group of the second satellite image of a predetermined size;
Each of the first block image belonging to the first block image group of the first satellite image and the first block image belonging to the first block image group of the second satellite image is selected from texture, vegetation index, and altitude. Classify based on one or more,
Based on the class of the first block image, the first block image is divided or combined with the first block image adjacent to the first block image, so that the second block image group of the first satellite image and Generating a second block image group of the second satellite image;
The second block image A belonging to the second block image group of the first satellite image and the second block image B belonging to the second block image group of the second satellite image are second higher than the first resolution. A satellite that generates a difference image between a region corresponding to the second block image A of the first satellite image having the resolution and a region corresponding to the second block image B of the second satellite image having the second resolution. Image processing method. In claim 10,
The first satellite image of the first resolution is a satellite image generated by reducing the resolution of the first satellite image of the second resolution,
The satellite image processing method, wherein the second satellite image having the first resolution is a satellite image generated by reducing the resolution of the second satellite image having the second resolution. In claim 10,
Among the first block images belonging to the first block image group, a satellite image that is a second block image by combining a first block image having a texture uniformity equal to or greater than a predetermined value with an adjacent first block image. Processing method. In claim 10,
A satellite image processing method for dividing a first block image of a different class into different sizes to form a second block image. In claim 10,
The third block image A of the first satellite image of the second resolution cut out based on the difference image and the third block image B of the second satellite image of the second resolution cut out based on the difference image. A satellite image processing method that performs image recognition applying deep learning to each. In claim 14,
In response to feedback from the user regarding the results of the image recognition of the third block image A and the third block image B, the teacher data stored in the teacher data database used for the image recognition is updated, and a predetermined number of times or more is updated. A satellite image processing method for updating a learning model of a recognizer used for the image recognition at a timing when the teacher data is updated.

Images