CN104657436A - Static tile pyramid parallel building method based on MapReduce - Google Patents
Static tile pyramid parallel building method based on MapReduce Download PDFInfo
- Publication number
- CN104657436A CN104657436A CN201510051204.2A CN201510051204A CN104657436A CN 104657436 A CN104657436 A CN 104657436A CN 201510051204 A CN201510051204 A CN 201510051204A CN 104657436 A CN104657436 A CN 104657436A
- Authority
- CN
- China
- Prior art keywords
- tile
- layer
- tiles
- pyramid
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 27
- 230000003068 static effect Effects 0.000 title claims abstract description 17
- 238000012545 processing Methods 0.000 claims abstract description 16
- 238000010276 construction Methods 0.000 claims description 28
- 238000005070 sampling Methods 0.000 claims description 4
- 238000006386 neutralization reaction Methods 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 abstract description 11
- 238000002474 experimental method Methods 0.000 description 29
- 239000003638 chemical reducing agent Substances 0.000 description 8
- 238000012952 Resampling Methods 0.000 description 5
- 230000002776 aggregation Effects 0.000 description 4
- 238000004220 aggregation Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000007726 management method Methods 0.000 description 4
- 238000012360 testing method Methods 0.000 description 4
- 238000011160 research Methods 0.000 description 3
- 238000003491 array Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 229920006395 saturated elastomer Polymers 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000013499 data model Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Landscapes
- Image Processing (AREA)
- Processing Or Creating Images (AREA)
Abstract
The invention provides a static tile pyramid parallel building method based on MapReduce, and belongs to the technical field of remote sensing image processing. The static tile pyramid parallel building method based on the MapReduce aims at fast performing batch remote sensing image parallel processing on the basis of a parallel calculation model of the MapReduce by aiming at the features of remote sensing images. The method is completed by four steps including Map tile processing, creation of Lmax layers of tiles, tile image generation and Reduce tile merging. The invention provides the improved static tile pyramid parallel building method based on the MapReduce, and the combination of fast parallel building of batch remote sensing image tile pyramid and tile pyramid large file generation is realized. Through structures of tile pyramid large files and HBase tile index tables, a fast retrieval model of global tile images is built.
Description
Technical Field
The invention belongs to the technical field of remote sensing image processing.
Background
The longitude and latitude subdivision grid model is a multi-level grid divided according to fixed longitude and latitude interval recursion, and has the advantages of simplicity, quickness and practicability. The traditional longitude and latitude subdivision grid model is mainly used for roughly expressing the spatial position of the earth surface, generally, the requirement on longitude is not high, and the number of layers for dividing grids is small. Typical mesh division models include World Geographic Reference System (Georef) and Global Area Reference System (GARS) of NGA. The subdivision grid model is derived from different application backgrounds, a longitude and latitude coordinate system is basically used as a division basis, subdivision levels are generally 3-4 layers, each layer has a complete coding system, and accurate geographical positioning is generally difficult to meet. The tile pyramid model is a currently recognized effective method for organizing and managing a large number of images. Since the tile pyramid is constructed by resampling to generate the low-resolution upper-layer image, the calculation amount is large and the time is long. With the rapid increase of the data volume of remote sensing images, the centralized processing mode has difficulty in meeting the processing requirements of the current remote sensing images. The subdivision tile pyramid is an effective means for uniformly managing mass images in the world, and can realize quick browsing of mass images in the same geographic coordinate system. Due to different projection modes of the image data, the tile pyramid models are different. Generally, the projection format of the image data needs to be determined reasonably according to application requirements.
Disclosure of Invention
The invention aims to provide a static tile pyramid parallel construction method based on MapReduce, which is used for quickly carrying out batch remote sensing image parallel processing aiming at the characteristics of remote sensing images and based on a MapReduce parallel calculation model.
The method comprises the following steps:
（1）map tile processing stage:calculating from a single remote-sensing imageAnd (3) the row and column ranges of the tiles of each layer of the generated tile pyramid are as follows:
determining the level range of the output tile according to the GTPM model and the space size and longitude and latitude range of the remote sensing imageWherein，Is the closest resolution in Table 3.2For each ofCalculate outLine and column ranges of each layer of tiles of tile pyramid in GTPM modelThe formula is as follows:
（3.1）
wherein,to round down; after enlargement or reductionLayer imageSize of pixelComprises the following steps:
（3.2）
wherein,in order to get the whole upwards,is composed ofThe pixel width and height of (d);
(2) creatingLayer tile task:
for theRelate toPer tile image of a layerComputing tilesIn thatPixel coordinate range of a layerComprises the following steps:
（3.3）
in pairSample generationTime, image dataContributing pixel rangeComprises the following steps:
（3.4）
wherein,is composed ofLongitude and latitude of the upper left corner pixel pointPixel coordinates on a layer; tileThe range of effective pixels (opaque pixel area) of (c) can be calculated as：
（3.5）
Wherein,is composed ofLongitude and latitude of the right lower corner pointPixel coordinates on a layer; determineNeutralizationEffective pixel region ofCorresponding pixel region(ii) a Construct the tile pyramidLayer tasks; then will beLayer tiles as inputPerforming the above steps, namelyTile task construction of a layer, at this point(ii) a The steps are circulated until a first layer of tiles are generated, and the creation of a tile pyramid task is realized;
(3) and (3) generation of tile images:
tile bottom layer LmaxTile image of
The tiles of the other levels are generated by sampling four tiles of the next level according to a bilinear interpolation method;
（4）reduce tile merging stage:
combining repeated areas between adjacent image frames in the Reduce stage; at this stage, for the situation of containing complete tiles, only one of the complete tiles is taken; when incomplete tiles are combined, the combination can be realized only by adopting a layer superposition principle; thus, mosaic and combination of tiles of each layer of the remote sensing image are completed.
The invention provides an improved static tile pyramid construction algorithm based on MapReduce batch remote sensing images, which realizes the combination of the rapid parallel construction of the batch remote sensing image tile pyramids and the generation of large files of the tile pyramids, saves the large files of the tile pyramids in a two-stage index form in an HDFS (Hadoop distributed file system), lays a foundation for the parallel storage of tile indexes managed by mass tile images and the rapid retrieval of an HBase tile index table, and realizes the instant opening and browsing of the remote sensing images. According to the invention, the rapid retrieval model of the global tile image is established through the structures of the tile pyramid large file and the HBase tile index table.
Drawings
FIG. 1 is a flow chart of the Map phase of the present invention;
FIG. 2 is a time consuming set-up of a tile pyramid for single and arrayed disks;
FIG. 3 is the CPU utilization of the Raid5 array and the single disk generation tile pyramid;
FIG. 4 is a time consuming batch tile pyramid construction at different node numbers;
FIG. 5 is the result of the Reduce phase experiment;
FIG. 6 is a graph showing a result of a tile image fast search experiment;
FIG. 7 is a graph of the second result of tile image fast search experiment;
FIG. 8 is a row-column encoding diagram for a third layer of tiles;
FIG. 9 is a Hilbert curve for the first five orders; wherein: (a) a first order Hilbert curve; (b) a second order Hilbert curve; (c) a third order Hilbert curve; (d) a fourth order Hilbert curve; (e) a fifth-order Hilbert curve;
fig. 10 is an image pyramid model based on Hilbert coding.
Detailed Description
The Global Tile Pyramid Model (GTPM) is a double-tower quadtree structure, and the image data adopts a Plate Carree projection. The east-west tiles of the first layer are root nodes of two quadtrees, and then each tile is divided into four parts on average to obtain the next layer of tiles, so that the double-tower quadtree structure is obtained. The tile sizes are all set to be 256 pixels multiplied by 256 pixels, the pyramid top layer is the first layer, and the number of rows and columns of tiles is 1 multiplied by 2.
First, theLevelNumber of rows and columns of tiles of a stageLevelThe relationship is as follows:。
the tiles below the first level are all 2 × 2 equal halves of the parent tile above, so the resolution is half that of the tile above, and the number of tiles in each level and the spatial resolution are shown in table 3.2.
TABLE 3.2 corresponding resolution of GTPM layers
| Hierarchy level | Line number | Number of rows | Resolution (longitude and latitude) | Equatorial resolution (meter) |
| 1 | 1 | 2 | 0.703125 | 78271.516964 |
| 2 | 2 | 4 | 0.3515625 | 39135.758482 |
| 3 | 4 | 8 | 0.17578125 | 19567.879241 |
| 4 | 8 | 16 | 0.087890625 | 9783.9396205 |
| 5 | 16 | 32 | 0.0439453125 | 4891.96981025 |
| 6 | 32 | 64 | 0.02197265625 | 2445.984905125 |
| 7 | 64 | 128 | 0.010986328125 | 1222.9924525625 |
| 8 | 128 | 256 | 0.0054931640625 | 611.49622628125 |
| 9 | 256 | 512 | 0.00274658203125 | 305.748113140625 |
| 10 | 512 | 1024 | 0.001373291015625 | 152.8740565703125 |
| 11 | 1024 | 2048 | 0.0006866455078125 | 76.43702828515625 |
| 12 | 2048 | 4096 | 0.00034332275390625 | 38.218514142578125 |
| 13 | 4096 | 8192 | 0.000171661376953125 | 19.1092570712890625 |
| 14 | 8192 | 16384 | 0.0000858306884765625 | 9.55462853564453125 |
| 15 | 16384 | 32768 | 0.00004291534423828125 | 4.777314267822265625 |
| 16 | 32768 | 65536 | 0.000021457672119140625 | 2.3886571339111328125 |
| 17 | 65536 | 131072 | 0.0000107288360595703125 | 1.19432856695556640625 |
| 18 | 131072 | 262144 | 0.00000536441802978515625 | 0.597164283477783203125 |
| … | … | … | … | .. |
The origin of the row-column coding agreed by the present invention is located at the top left corner of each layer of tiles, and the row-column coding of the third layer of tiles is shown in fig. 8.
In general, to improve the spatial aggregation of tile image data storage, it is necessary to apply a spatial filling curve to the tile storage of the image pyramid. At present, the common space filling curves include a line sequence, a Peano curve, a Hilbert curve and the like. The Hilbert curve, which originates from the classical Peano curve cluster and is the one of the best spatial aggregations among the currently known fill curves, is shown in fig. 9 as the first five-step Hilbert plot.
The invention adopts 0, 1, 2 and 3 to sequentially represent the sequence of Hilbert curves passing through a 2 multiplied by 2 basic type unit. East and west hemispheres of the first layer are coded with "1" and "0", respectively. When the tile indexes are stored in the HBase database, each tile corresponds to a quaternary Hilbert coded string (abbreviated as "Hcode"), such as "0132", and fig. 10 shows the pyramid model of the tile according to the present invention. The Hcode-based organization tile has the characteristics that:
(1) the Hcode character length of the tile is equal to the stage number;
(2) the spatial positions of the adjacent tiles on the Hcode are also adjacent; tiles that are spatially adjacent, Hcode is also generally adjacent. If the data are stored according to the sequence of the Hcodes, the excellent space aggregation property of the Hilbert curve can be fully utilized;
(3) the Hcode can correspond to a main key RowKey of the HBase table, and is favorable for quickly retrieving tile index data from the HBase table.
After the remote sensing image completes tile resampling and Block cutting tasks in a Map stage, adjacent tiles are sent to the same Reduce node according to the Hcode sequencing, tile sets are combined into a Block tile set in a Reducer according to the Hcode sequence, and therefore the tile image is stored according to a Hilbert coding curve, the space aggregation of the Hilbert curve can be fully utilized, and the reading speed of the tiles in batches can be accelerated during retrieval.
The remote sensing images for constructing the static tile pyramid are all orthoimages which are preprocessed and accord with the Plate Carree projection of a 2000 national geodetic coordinate system. When the static tile pyramid is constructed in parallel, because the overlapping rate of the adjacent image frames is large, merging processing is needed to eliminate repeated tile images and reduce data redundancy. The invention combines a plurality of acquired continuous remote sensing images to generate a single image pyramid file, and eliminates repeated image data in the combining process. The static tile pyramid parallel construction algorithm based on MapReduce mainly comprises two stages of Map and Reduce. The specific algorithm flow is as follows: firstly, the Client stores the remote sensing image in the HDFS, then submits a tile pyramid construction task, and starts to construct the tile pyramid in parallel. And the MapReduce allocates the remote sensing image tile pyramid construction task to the Mapper node according to the task allocation strategy, so that parallel processing based on cloud is realized. In the Map stage, each Mapper node allocated to a task reads the allocated remote sensing image from the HDFS, and the local image is processed in parallel by adopting a multithreading technology to generate a tile pyramid corresponding to the image of the Map frame. Each tile generated is grouped in the MapReduce framework and then sent to the corresponding Reducer node. In the Reduce stage, after the Reducer node completes the tile combination, the tiles are organized into a Block tile set file according to the Hcode sequence. And uploading the information of each Block tile set file to a NameNode of the HDFS, extracting tile index areas in the Block tile set files, and combining to generate a tile pyramid index file of the whole remote sensing area. Therefore, parallel construction of the tile pyramid large file ImageFile of the whole remote sensing area is realized. The tiles of the static tile pyramid are of uniform size, all 256 pixels by 256 pixels.
Through the processing of the Map stage and the Reduce stage, resampling, cutting and embedding of tiles are completed on a plurality of remote sensing images, and finally a single tile pyramid large file of a batch of remote sensing images is generated in a combined mode.
The main function of the Map stage is to resample the remote sensing image and to cut the remote sensing image into blocks to generate a tile pyramid corresponding to the image of the Map. The input Key is M containing the geographic position and the resolution information of the remote sensing image, and the Value is the remote sensing image data corresponding to M. In order to realize local calculation of the remote sensing image, the remote sensing image is read from the HDFS to the local, and then the remote sensing image is processed to generate a tile pyramid. And re-sampling the remote sensing image to ensure that the resolution of the amplified or reduced image is equal to that of the nearest level. The target resolution of resampling is set asCorresponding to a level ofThen, the magnification ratio from the remote sensing image to the target image is:（as remote sensing imagesThe resolution of (d). The steps of resampling and dicing the image according to the magnification ratio and the geographic position and resolution information of the remote sensing image, and generating the tile pyramid algorithm at the Map stage are shown in fig. 1.
（1）Map tile processing stage:calculating from a single remote-sensing imageAnd (3) the row and column ranges of the tiles of each layer of the generated tile pyramid are as follows:
determining the level range of the output tile according to the GTPM model and the space size and longitude and latitude range of the remote sensing imageWherein，Is the closest resolution in Table 3.2For each ofCalculate outLine and column ranges of each layer of tiles of tile pyramid in GTPM modelThe formula is as follows:
（3.1）
wherein,to round down; after enlargement or reductionLayer imageSize of pixelComprises the following steps:
（3.2）
wherein,in order to get the whole upwards,is composed ofThe pixel width and height of (d);
(2) creatingLayer tile task:
for theRelate toPer tile image of a layerComputing tilesIn thatPixel coordinate range of a layerComprises the following steps:
（3.3）
in pairSample generationTime, image dataContributing pixel rangeComprises the following steps:
（3.4）
wherein,is composed ofLongitude and latitude of the upper left corner pixel pointPixel coordinates on a layer; tileThe range of effective pixels (opaque pixel area) of (c) can be calculated as：
（3.5）
Wherein,is composed ofLongitude and latitude of the right lower corner pointPixel coordinates on a layer; determineNeutralizationEffective pixel region ofCorresponding pixel region(ii) a Construct the tile pyramidLayer tasks; then will beLaminated tileUsing the sheet as inputPerforming the above steps, namelyTile task construction of a layer, at this point(ii) a The steps are circulated until a first layer of tiles are generated, and the creation of a tile pyramid task is realized;
(3) and (3) generation of tile images:
tile bottom layer LmaxTile image of
The tiles of the other levels are generated by sampling four tiles of the next level according to a bilinear interpolation method;
（4）reduce tile merging stage:
combining repeated areas between adjacent image frames in the Reduce stage; at this stage, for the situation of containing complete tiles, only one of the complete tiles is taken; when incomplete tiles are combined, the combination can be realized only by adopting a layer superposition principle; thus, mosaic and combination of tiles of each layer of the remote sensing image are completed. After merging, merging the tiles under the nodes into a tile pyramid large file.
And extracting the tile indexes of the large file of the tile pyramid and writing the tile indexes into an HBase tile index table, and the Client can quickly search massive remote sensing images by searching the HBase index table and reading the tile images in the Block.
And (3) verification:
the experiment of the invention is carried out on a computer cluster, so the performance of the experiment is determined by the hardware and software configuration level of the cluster computer.
5.1.1 hardware Environment
The experimental computing cluster comprises 1 control node and 7 computing nodes, and the computer configurations of the two nodes are shown in table 5.1. The network data switches in the experimental environment are all gigabit switches.
TABLE 5.1 Cluster computer configuration
5.1.2 software Environment and image data
The software environment mainly comprises a node operating system, a virtual machine, a Hadoop platform and the like, and is specifically shown in a 5.2 table. In order to improve the efficiency of inquiring the remote sensing image metadata, the invention adopts a MySQL database to store the metadata information of the tile pyramid big file. The development tools used by Client are Microsoft Visual Studio 2008, GDAL, and the like. The experimental image data are shown in table 5.3.
TABLE 5.2 software Environment
TABLE 5.3 image data conditions of the test environments
5.2 Rapid construction experiment of tile pyramid
In order to verify the effectiveness of the algorithm, the method provided by the invention performs an experiment on the construction of the static tile pyramid. The experiment adopts a true color image of a certain area with 2.14 m resolution ratio shot by a certain remote sensing satellite, and the image storage format is GeoTiff. The static tile pyramid construction method experiment is divided into a Map stage experiment and a Reduce stage experiment, and the calculation efficiency and the resource utilization capacity of the static tile pyramid parallel construction algorithm based on MapReduce are mainly verified.
5.2.1 Map phase experiment
In the Map stage, reading and writing the remote sensing image corresponding to each Mapper node into a local disk by each Mapper node, and then performing local calculation on the remote sensing image by the Mapper node to generate a tile pyramid corresponding to the Map sheet. In the Map stage of the invention, the tile pyramid is independently constructed in parallel at each Mapper node, so the algorithm efficiency is restricted by the calculation efficiency under a single node and the parallel construction efficiency under a cluster. In order to test the effectiveness of the algorithm, two aspects of experiments need to be carried out, namely a tile pyramid construction experiment of a single remote sensing image under a single node, and a tile pyramid parallel construction experiment of a plurality of remote sensing images under a plurality of nodes.
(1) Single-node single remote sensing image tile pyramid construction
The size of the experimental remote sensing image is 1.18GB, the space size is 40632 pixel multiplied by 33301 pixel, the tile pyramid is constructed by adopting the multithreading technology, so that the multithreading efficiency and the resource utilization capacity of the static tile pyramid construction algorithm based on MapReduce are verified, and the experimental software and hardware environment is detailed in the software and hardware environment.
And storing the remote sensing image on a node computer, and then starting a tile pyramid construction program. Firstly, according to the space size and the geographic information of the remote sensing image, a tile pyramid is calculatedA range of rows and columns of tiles of a layer. Then, the Map stage tile task construction algorithm is adopted to establishTile output tasks of a layer, parallel execution of tile tasks using multiple threads, outputLayer tiles. Finally, the same basic unit is placed belowThe tiles of a layer are merged and resampled into a tile of the previous layer, e.g., tiles "05 _ 10230", "05 _ 10231", "05 _ 10232", and "05 _ 10233" are merged and sampled into a tile of the previous layer "04 _ 1023" according to the Hilbert curve position.
Because the basic units are independent, the previous layer of tiles can be generated by adopting multithreading, and the previous layer of tiles are generated in a circulating mode continuously until the first layer of tiles is finished. The experiment is completed on a single 3TB mechanical disk and 8 3TB mechanical disk Raid5 arrays by adopting different thread numbers respectively, and the time consumption conditions under different thread numbers are recorded.
FIG. 2 is a time consuming scenario for generating a tile pyramid of a remotely sensed image using 3TB mechanical disks with single and 8 block Raid5 arrays, respectively, as data storage. As can be seen, as the number of operation threads increases, the time consumption is reduced, but the time consumption of the Raid5 disk array is less than that of a single disk, which is mainly caused by the fact that the I/O of the Raid5 array is larger than that of the single disk. In the case of a single disk, when the number of threads reaches 7, the time consumption is 39.16 seconds, and then as the number of threads increases, the time consumption is not reduced significantly, which indicates that the processing capacity in the case of a single disk has been maximized, and since the disk I/O of a single disk is basically saturated in the case of 7 threads, the disk I/O is maintained at about 120 MB/S. Combining the CPU utilization of the single disk and Raid5 array of fig. 3, it can be seen that the CPU utilization of the single disk for 7 threads is saturated due to disk I/O limitations. And the influence of thread scheduling and the like, so that the time for generating the tile pyramid is stable when 7 threads exist. Then, as the number of threads increases, the total consumed time under the condition of a single disk remains stable, and when the number of threads exceeds the number of CPU cores 16, CPU resources are contended among the threads, so that the consumed time is increased.
From the experimental analysis, the tile pyramid algorithm can maximally utilize computer resources and quickly construct the tile pyramid.
(2) Tile pyramid is constructed in parallel to many remote sensing images under multinode
After the remote sensing images are read and written into the corresponding Mapper nodes, the tile pyramid is constructed in parallel by utilizing the improved algorithm of the invention by respectively adopting different node numbers for the 5 remote sensing images. And each node is under a Raid5 disk array, and a tile pyramid of a single remote sensing image is constructed in parallel by adopting 16 threads. FIG. 4 is a relationship between the number of cluster nodes in the experiment and the total time spent building the tile pyramid.
According to the experimental result graph, when the number of the nodes is smaller than the number of the remote sensing image graphs, the average task amount of the nodes is larger than 1, and the total time consumption is long. When the number of the nodes is larger than or equal to the image map number, each remote sensing image generates a tile pyramid by one node, the total time consumption is small, and the calculation is fast. Therefore, when the tile pyramid is generated by cluster processing, the operation time can be shortest under the condition that the number of cluster nodes is greater than the number of remote sensing image amplitudes to be processed.
5.2.2 Reduce phase experiment
After the construction of the tile pyramids of the 5 remote sensing images is completed, the generated tiles are sent to a Reducer to organize the tile images into a Block tile set file according to the Hcode sequence. The experiment yielded a total of 19 blocks of the Block tile set file, with a final Block size of 38.4MB, totaling 1190.4 MB. The experiment tests the relation between the consumed time and the number of reducers in the Reduce stage under different numbers of reducers (the number of nodes). The test results are shown in FIG. 5, which shows that the time required for the number of reducers to be 1 is 29 minutes and 22 seconds (1762 seconds). The time consumption decreases rapidly as the number of nodes increases, and when the number of nodes increases to 7, the time consumption is only 150 seconds. According to the experimental result, the time consumption of the Reduce stage can be greatly reduced by increasing the number of reducers in the Reduce stage. Therefore, the Block tile file set construction in the Reduce stage can be completed quickly in a cluster environment.
Tile image fast retrieval experiment
In order to verify the role of the ImageTable in image retrieval, the invention designs two experiments to respectively verify the retrieval function and the retrieval efficiency of the ImageTable.
Experiment one: the tile pyramid image of the experiment is directly stored by using the method of the invention and the relational database SQL Server 2005 under Windows environment, and the experimental data is shown in Table 5.4. And respectively searching the image in the same area, wherein the size of the search area is 20 tiles, and the image in the area is read once and then returned during searching. The experimental result is shown in fig. 6, the abscissa is the number of pyramid files of different images stored in the system, and in the ordinate, the ordinate data represents the time consumed for directly searching the SQL Server 2005 database and the method of the present invention.
TABLE 5.4 number of pyramid files and data size of image
| Image pyramid file number (number) | 11 | 58 | 107 | 178 | 276 |
| Data volume (GB) | 12.8 | 71.2 | 129.8 | 212.3 | 333.4 |
The reasons for this experimental result are mainly: the ImageTable stores the data of each tile index, and the corresponding tile index is retrieved, so that the corresponding Block tile image file in the HDFS can be directly read. Although the comparison method does not need to read the tiles in the HDFS, the comparison method is limited by the SQL Server image data management capability of the relational database, so the ImageTable for managing massive remote sensing images based on Hadoop has higher image retrieval efficiency than the relational database.
Experiment two: since the tile needs to be retrieved from the ImageTable again to obtain the information of the image file of the next region when the method reads the image set of the cross-region, the cross-region retrieval may affect the image roaming speed (the tile image in the image roaming experiment is only read into the memory and is not displayed). Experiment two compares the method of the present invention with a method of directly storing tile images in an HBase table (direct storage method for short). During the experiment, different numbers of tiles are roamed, and the single area and cross-area roaming effects of the direct storage method and the method are recorded respectively, and the result is shown in fig. 7.
It can be seen that as the number of roaming tiles increases, the single area roaming using the method of the present invention takes the least time, the roaming across multiple areas is centered, and the direct storage method takes the most time.
The effectiveness of the key technology research for managing the mass remote sensing images based on Hadoop is further verified by managing the image data of a certain scale through the experiment, and the potential capability and the advantage of managing the mass remote sensing images by utilizing cloud computing are shown.
Aiming at the problems of low retrieval efficiency, poor sharing capability, incapability of realizing integrated management and the like faced by the management of mass remote sensing images, the invention adopts an open source cloud computing platform Hadoop to manage the mass remote sensing images, researches some key technologies, and obtains the following research results:
1. the method comprises the steps of analyzing and summarizing a Hadoop platform, analyzing main characteristics, a system structure and a main flow of the HDFS in detail, introducing a structure model and an operation flow of a parallel computing model MapReduce in detail, and summarizing the system structure and a data model of an HBase database.
2. For remote sensing images conforming to the Plate Carree projection format, the invention provides a global subdivision tile pyramid model based on equal longitude and latitude grid division, and establishes a global tile image Hcode code by utilizing a Hilbert curve. On the basis of the tile pyramid model, a static tile pyramid construction algorithm suitable for MapReduce parallel computing is improved and provided. The algorithm can give full play to the cluster power, realize rapid parallel processing of batch remote sensing images and generate large files of the tile pyramid. For the remote sensing image needing to be opened and browsed instantly, the invention improves and provides a dynamic tile pyramid construction algorithm, and realizes seamless combination of remote sensing image browsing and tile pyramid construction.
3. In order to quickly search massive tile images, a massive tile image management model combining a tile pyramid large file and an HBase tile index table is improved and provided. The model can realize the quick retrieval and reading of the tile images, thereby achieving the purpose of integrated management and sharing of the remote sensing images. The calculation method of the Hcode is improved, a table look-up based fast calculation method of the Hcode and a parallel storage method of a Block tile set are provided, and the calculation efficiency of the Hcode and the storage efficiency of the tiles are improved.
Claims (1)
1. A static tile pyramid parallel construction method based on MapReduce is characterized by comprising the following steps: the method comprises the following steps:
（1）map tile processing stage:calculating from a single remote-sensing imageAnd (3) the row and column ranges of the tiles of each layer of the generated tile pyramid are as follows:
according to the GTPM modelDetermining the range of output tile levels according to the space size and longitude and latitude range of remote sensing imageWherein，To the nearest resolutionFor each ofCalculate outLine and column ranges of each layer of tiles of tile pyramid in GTPM modelThe formula is as follows:
（3.1）
wherein,to round down; after enlargement or reductionLayer imageSize of pixelComprises the following steps:
（3.2）
wherein,in order to get the whole upwards,is composed ofThe pixel width and height of (d);
(2) creatingLayer tile task:
for theRelate toPer tile image of a layerComputing tilesIn thatPixel coordinate range of a layerComprises the following steps:
（3.3）
in pairSample generationTime, image dataContributing pixel rangeComprises the following steps:
（3.4）
wherein,is composed ofLongitude and latitude of the upper left corner pixel pointPixel coordinates on a layer; tileThe range of effective pixels (opaque pixel area) of (c) can be calculated as：
（3.5）
Wherein,is composed ofLongitude and latitude of the right lower corner pointPixel coordinates on a layer; determineNeutralizationEffective pixel region ofCorresponding pixel region(ii) a Construct the tile pyramidLayer tasks; then will beLayer tiles as inputPerforming the above steps, namelyLayer of tilesService construction at this point(ii) a The steps are circulated until a first layer of tiles are generated, and the creation of a tile pyramid task is realized;
(3) and (3) generation of tile images:
tile bottom layer LmaxTile image of
The tiles of the other levels are generated by sampling four tiles of the next level according to a bilinear interpolation method;
（4）reduce tile merging stage:
combining repeated areas between adjacent image frames in the Reduce stage; at this stage, for the situation of containing complete tiles, only one of the complete tiles is taken; when incomplete tiles are combined, the combination can be realized only by adopting a layer superposition principle; thus, mosaic and combination of tiles of each layer of the remote sensing image are completed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510051204.2A CN104657436B (en) | 2015-02-02 | 2015-02-02 | Static tile pyramid parallel constructing method based on MapReduce |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510051204.2A CN104657436B (en) | 2015-02-02 | 2015-02-02 | Static tile pyramid parallel constructing method based on MapReduce |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104657436A true CN104657436A (en) | 2015-05-27 |
| CN104657436B CN104657436B (en) | 2019-01-25 |
Family
ID=53248564
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510051204.2A Expired - Fee Related CN104657436B (en) | 2015-02-02 | 2015-02-02 | Static tile pyramid parallel constructing method based on MapReduce |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104657436B (en) |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105303535A (en) * | 2015-11-15 | 2016-02-03 | 中国人民解放军空军航空大学 | Global subdivision pyramid model based on wavelet transformation |
| CN105469434A (en) * | 2015-12-31 | 2016-04-06 | 中南林业科技大学 | Method of browsing large remote sensing image in Android system |
| CN105956165A (en) * | 2016-05-20 | 2016-09-21 | 江苏得得空间信息科技有限公司 | Tile type large file organization storage method of three-dimensional model data |
| CN106528699A (en) * | 2016-11-02 | 2017-03-22 | 北京航天泰坦科技股份有限公司 | IO multi-thread computing method and system for fast building image pyramid |
| CN106991143A (en) * | 2017-03-22 | 2017-07-28 | 苏州中科图新网络科技有限公司 | Multi-layer image file, generation method and device, read method and device |
| CN108932303A (en) * | 2018-06-12 | 2018-12-04 | 中国电子科技集团公司第二十八研究所 | A kind of distribution visual remote sensing image Detection dynamic target and analysis system |
| CN109859094A (en) * | 2019-01-25 | 2019-06-07 | 中国测绘科学研究院 | A kind of pyramidal fast construction method of timing image tile |
| CN109933565A (en) * | 2019-01-31 | 2019-06-25 | 湖北省基础地理信息中心（湖北省北斗卫星导航应用技术研究院） | A kind of rapid section method and system of multiple dimensioned mass remote sensing image |
| CN110347499A (en) * | 2019-06-13 | 2019-10-18 | 武汉大学 | A kind of remote sensing image tile generates and the method for deployed in real time |
| CN110555119A (en) * | 2019-08-27 | 2019-12-10 | 成都数之联科技有限公司 | Unmanned aerial vehicle remote sensing image slicing method and system under real-time scene |
| CN110706341A (en) * | 2019-09-17 | 2020-01-17 | 广州市城市规划勘测设计研究院 | High-performance rendering method and device of city information model and storage medium |
| CN111782742A (en) * | 2020-06-06 | 2020-10-16 | 中国科学院电子学研究所苏州研究院 | Large-scale geographic space data oriented storage and retrieval method and system thereof |
| CN113012094A (en) * | 2020-11-30 | 2021-06-22 | 中国科学院空天信息创新研究院 | Processing method of satellite remote sensing data and related device |
| CN113706556A (en) * | 2021-08-20 | 2021-11-26 | 珠江水利委员会珠江水利科学研究院 | Rapid detection and replacement method for image slice edge fragments |
| CN113793411A (en) * | 2021-05-24 | 2021-12-14 | 北京建筑大学 | Data organization and dynamic expression method based on linear reference |
| CN114201726A (en) * | 2020-09-18 | 2022-03-18 | 深圳先进技术研究院 | Convolution operation optimization method, system, terminal and storage medium |
| CN114898026A (en) * | 2022-05-10 | 2022-08-12 | 北京领为军融科技有限公司 | Dynamic loading and unloading method for landscape based on position and sight |
| CN115470366A (en) * | 2022-08-31 | 2022-12-13 | 湖南省第二测绘院 | Tile-based remote sensing image storage method and system |
| CN116188997A (en) * | 2023-04-24 | 2023-05-30 | 中铁城际规划建设有限公司 | Remote sensing image element data processing method and system |
| CN117271691A (en) * | 2023-10-07 | 2023-12-22 | 扬州万方科技股份有限公司 | Construction method, system and computer readable medium for tile map service based on domestic platform |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103455624A (en) * | 2013-09-16 | 2013-12-18 | 湖北文理学院 | Implement method of lightweight-class global multi-dimensional remote-sensing image network map service |
| CN103796018A (en) * | 2014-01-14 | 2014-05-14 | 齐齐哈尔大学 | Remote-sensing image real-time compression and progressive transmission system |
| CN104091301A (en) * | 2014-06-30 | 2014-10-08 | 江苏华大天益电力科技有限公司 | Tile pyramid parallel construction method based on MapReduce |
-
2015
- 2015-02-02 CN CN201510051204.2A patent/CN104657436B/en not_active Expired - Fee Related
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103455624A (en) * | 2013-09-16 | 2013-12-18 | 湖北文理学院 | Implement method of lightweight-class global multi-dimensional remote-sensing image network map service |
| CN103796018A (en) * | 2014-01-14 | 2014-05-14 | 齐齐哈尔大学 | Remote-sensing image real-time compression and progressive transmission system |
| CN104091301A (en) * | 2014-06-30 | 2014-10-08 | 江苏华大天益电力科技有限公司 | Tile pyramid parallel construction method based on MapReduce |
Non-Patent Citations (1)
| Title |
|---|
| 刘义等: "利用MapReduce进行批量遥感影像瓦片金字塔构建", 《武汉大学学报 信息科学版》 * |
Cited By (29)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105303535A (en) * | 2015-11-15 | 2016-02-03 | 中国人民解放军空军航空大学 | Global subdivision pyramid model based on wavelet transformation |
| CN105469434A (en) * | 2015-12-31 | 2016-04-06 | 中南林业科技大学 | Method of browsing large remote sensing image in Android system |
| CN105469434B (en) * | 2015-12-31 | 2018-10-30 | 中南林业科技大学 | A method of browsing large-scale remote sensing image in android system |
| CN105956165A (en) * | 2016-05-20 | 2016-09-21 | 江苏得得空间信息科技有限公司 | Tile type large file organization storage method of three-dimensional model data |
| CN105956165B (en) * | 2016-05-20 | 2019-01-18 | 江苏得得空间信息科技有限公司 | A kind of big file storage organization method of the tile type of three-dimensional modeling data |
| CN106528699A (en) * | 2016-11-02 | 2017-03-22 | 北京航天泰坦科技股份有限公司 | IO multi-thread computing method and system for fast building image pyramid |
| CN106991143B (en) * | 2017-03-22 | 2019-07-19 | 苏州中科图新网络科技有限公司 | Multi-layer image file, generation method and device, read method and device |
| CN106991143A (en) * | 2017-03-22 | 2017-07-28 | 苏州中科图新网络科技有限公司 | Multi-layer image file, generation method and device, read method and device |
| CN108932303A (en) * | 2018-06-12 | 2018-12-04 | 中国电子科技集团公司第二十八研究所 | A kind of distribution visual remote sensing image Detection dynamic target and analysis system |
| CN108932303B (en) * | 2018-06-12 | 2022-02-08 | 中国电子科技集团公司第二十八研究所 | Distributed dynamic target detection and analysis system for visible light remote sensing image |
| CN109859094A (en) * | 2019-01-25 | 2019-06-07 | 中国测绘科学研究院 | A kind of pyramidal fast construction method of timing image tile |
| CN109859094B (en) * | 2019-01-25 | 2021-02-09 | 中国测绘科学研究院 | Rapid construction method of time sequence image tile pyramid |
| CN109933565A (en) * | 2019-01-31 | 2019-06-25 | 湖北省基础地理信息中心（湖北省北斗卫星导航应用技术研究院） | A kind of rapid section method and system of multiple dimensioned mass remote sensing image |
| CN110347499A (en) * | 2019-06-13 | 2019-10-18 | 武汉大学 | A kind of remote sensing image tile generates and the method for deployed in real time |
| CN110347499B (en) * | 2019-06-13 | 2021-04-16 | 武汉大学 | Method for generating and deploying remote sensing image tiles in real time |
| CN110555119A (en) * | 2019-08-27 | 2019-12-10 | 成都数之联科技有限公司 | Unmanned aerial vehicle remote sensing image slicing method and system under real-time scene |
| CN110555119B (en) * | 2019-08-27 | 2022-05-13 | 成都数之联科技股份有限公司 | Unmanned aerial vehicle remote sensing image slicing method and system under real-time scene |
| CN110706341A (en) * | 2019-09-17 | 2020-01-17 | 广州市城市规划勘测设计研究院 | High-performance rendering method and device of city information model and storage medium |
| CN111782742A (en) * | 2020-06-06 | 2020-10-16 | 中国科学院电子学研究所苏州研究院 | Large-scale geographic space data oriented storage and retrieval method and system thereof |
| CN114201726A (en) * | 2020-09-18 | 2022-03-18 | 深圳先进技术研究院 | Convolution operation optimization method, system, terminal and storage medium |
| CN113012094A (en) * | 2020-11-30 | 2021-06-22 | 中国科学院空天信息创新研究院 | Processing method of satellite remote sensing data and related device |
| CN113793411A (en) * | 2021-05-24 | 2021-12-14 | 北京建筑大学 | Data organization and dynamic expression method based on linear reference |
| CN113706556A (en) * | 2021-08-20 | 2021-11-26 | 珠江水利委员会珠江水利科学研究院 | Rapid detection and replacement method for image slice edge fragments |
| CN113706556B (en) * | 2021-08-20 | 2022-06-10 | 珠江水利委员会珠江水利科学研究院 | Rapid detection and replacement method for image slice edge fragments |
| CN114898026A (en) * | 2022-05-10 | 2022-08-12 | 北京领为军融科技有限公司 | Dynamic loading and unloading method for landscape based on position and sight |
| CN115470366A (en) * | 2022-08-31 | 2022-12-13 | 湖南省第二测绘院 | Tile-based remote sensing image storage method and system |
| CN116188997A (en) * | 2023-04-24 | 2023-05-30 | 中铁城际规划建设有限公司 | Remote sensing image element data processing method and system |
| CN117271691A (en) * | 2023-10-07 | 2023-12-22 | 扬州万方科技股份有限公司 | Construction method, system and computer readable medium for tile map service based on domestic platform |
| CN117271691B (en) * | 2023-10-07 | 2024-06-14 | 扬州万方科技股份有限公司 | Construction method, system and computer readable medium for tile map service based on domestic platform |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104657436B (en) | 2019-01-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104657436B (en) | Static tile pyramid parallel constructing method based on MapReduce | |
| CN111291016B (en) | Hierarchical hybrid storage and indexing method for massive remote sensing image data | |
| CN105786942B (en) | A kind of geography information storage system based on cloud platform | |
| Eldawy et al. | The era of big spatial data | |
| Cary et al. | Experiences on processing spatial data with mapreduce | |
| Liang et al. | A visualization-oriented 3D method for efficient computation of urban solar radiation based on 3D–2D surface mapping | |
| CN103995861B (en) | A kind of distributed data device based on space correlation, method and system | |
| CN107515952B (en) | Method and system for point cloud data storage, parallel computing and real-time retrieval | |
| US10055808B1 (en) | Distributed and parallelized visualization framework | |
| Hongchao et al. | Distributed data organization and parallel data retrieval methods for huge laser scanner point clouds | |
| CN103281376A (en) | Method for automatic caching construction of massive timing sequence remote-sensing images in cloud environment | |
| Kumar et al. | PIDX: Efficient parallel I/O for multi-resolution multi-dimensional scientific datasets | |
| CN101339570A (en) | Efficient distributed organization and management method for mass remote sensing data | |
| US10895946B2 (en) | Systems and methods for using tiled data | |
| CN103793442A (en) | Spatial data processing method and system | |
| CN103744998A (en) | Hierarchical-division based storage and organization method for large-scale space vector data | |
| Zhang | Speeding up large-scale geospatial polygon rasterization on GPGPUs | |
| Nguyen et al. | B-EagleV: visualization of big point cloud datasets in civil engineering using a distributed computing solution | |
| Guo et al. | An efficient data organization and scheduling strategy for accelerating large vector data rendering | |
| CN107679127A (en) | Point cloud information parallel extraction method and its system based on geographical position | |
| CN117808980A (en) | Map grid picture generation method, system, equipment and storage medium | |
| CN111061806B (en) | Storage method and networked access method for distributed massive geographic tiles | |
| CN115393530B (en) | Massive three-dimensional model single rendering method, storage medium and electronic equipment | |
| Li et al. | An improved distributed storage model of remote sensing images based on the HDFS and pyramid structure | |
| Palmer et al. | Efficient data IO for a parallel global cloud resolving model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20190125 Termination date: 20200202 |