CN104091301B - A kind of tile pyramid parallel constructing method based on MapReduce - Google Patents
A kind of tile pyramid parallel constructing method based on MapReduce Download PDFInfo
- Publication number
- CN104091301B CN104091301B CN201410305679.5A CN201410305679A CN104091301B CN 104091301 B CN104091301 B CN 104091301B CN 201410305679 A CN201410305679 A CN 201410305679A CN 104091301 B CN104091301 B CN 104091301B
- Authority
- CN
- China
- Prior art keywords
- tile
- image
- data
- layer
- mapreduce
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Abstract
The invention provides a kind of tile pyramid parallel constructing method based on MapReduce, comprise the following steps：Step 10：Pyramid bottom is used as using ultimate resolution image；Step 20：Piecemeal is carried out to bottom, and the image blocks being divided into are encoded；Step 30：Image blocks data input after block encoding in step 20 is generated to new one layer of tile data into MapReduce model；Step 40：MapReduce model is often taken turns to new one layer of tile data of generation successively as the input data of next round MapReduce model, more last layer tile data is generated, terminated until pyramid model is built.The present invention is not only with good expansibility, and the parallel structure pyramid model on computer cluster, has higher efficiency for mass data.
Description
Technical field
The pyramidal method of tile, more particularly to a kind of tile gold based on MapReduce are built the present invention relates to one kind
Word tower parallel constructing method.
Background technology
Raster data is the row and column arrangement by grid cell, the array data with different gray scales or color, general work
Loaded for the bottom figure layer of map, serve as the background image of generalized information system.In order to shorten GIS server data access time, simultaneously
The problem of solving multiresolution, generally carries out tile piecemeal by different resolution and zoom level by raster data, constructs tile
Pyramid, the image blocks obtained by piecemeal are called one piece " tile " of original image.Pyramid model is a kind of static multiresolution layer
Secondary model, it is using the redundant storage of geography information as cost, to exchange the faster response speed of GIS platform for.Pyramid model is straight
The data that different resolution is provided are connect without carrying out real-time resampling, such client is when asking raster data, server
End, which is provided, can meet its part tile for showing demand, rather than the whole panoramic picture of transmission, so as to avoid largely being not required to close
The loading of the key element of note, transmit and render, mitigate the pressure of client rendering module significantly, reduce the load of the network bandwidth, from
And improve the overall efficiency of system.
Current multi-resolution pyramid (multi-resolution pyramid) oneself turn into processing big data quantity grid number
According to the conventional methods of collection.Multi-resolution pyramid keeps fixed resolution ratio, typically 2 between layer by layer:1, adopted with layer data
Split with the tile of equal sizes.Famous Google Earth are exactly to employ many pyramidal forms of resolution to store
Its mass remote sensing image, the supports of public users is obtained because the operation of simple and fast and quick network browsing are shown.
NASA World Wind, Microsoft Virtual Earth etc. also use this mode.
With dimensional information's basic installation construction and the fast development of Spatial data capture technology, spatial data scale is more next
Bigger, raster data just develops as a kind of important spatial data towards high-resolution direction.High-resolution means greatly
Data volume, for the Raster Images data of areal different resolution, resolution ratio is higher, and data volume is bigger, between the two not
It is simple linearly increasing, but exponentially increases again.Traditional pyramid construction algorithm based on unit is used for magnanimity grid
Image data builds pyramid, not only time length, inefficiency, and single node also easily becomes limitation bottleneck.
The content of the invention
Goal of the invention：In view of the above-mentioned deficiencies in the prior art, it is an object of the present invention to which provide one kind has for mass data
The tile pyramid parallel constructing method based on MapReduce of higher efficiency.
Technical scheme：The invention provides a kind of tile pyramid parallel constructing method based on MapReduce, including with
Lower step：
Step 10：Pyramid bottom is used as using ultimate resolution image；
Step 20：Piecemeal is carried out to bottom, and the image blocks being divided into are encoded；
Step 30：Image blocks data input after block encoding in step 20 is generated new one into MapReduce model
The tile data of layer；
Step 40：New one layer of the tile data that MapReduce model is often taken turns to generation successively is used as next round
The input data of MapReduce model, generates more last layer tile data, terminates until pyramid model is built.
Further, the method for partition in the step 20 is：The slit mode of quaternary tree carries out piecemeal to image.Using this
Slit mode quaternary tree index easy to use is planted quickly to be positioned.
Further, the coding method in the step 20 is：To piecemeal by the way of based on grid and Hilbert codings
Into image tile encoded；The numbering of each image tile by zoom level Z, Hilbert encode, tile row number and
Tile line number is constituted.Coding based on grid can quickly and easily calculate corresponding tile file according to given coordinate range；
And Hilbert codings have good Clustering Effect, when Hilbert permutation codes are adjacent or close, its corresponding extraterrestrial target
When certain adjacent or close, image tile is stored based on Hilbert codings, can keep image tile file after subdivision
Certain ability of aggregation.
Further, the method for every layer of tile data generation is：
Step 401：Each image tile file Z_H_C_R.dat of place layer produces a series of key assignments by Map functions
To Key->Value, wherein key are last layer father's image tiles code, subtract 1 by the zoom level of this layer of tile file,
No. Hilbert divided by 4 round, row, column number respectively divided by 2 round acquisition；Value is made up of reference number of a document i and tile file；Its
Middle reference number of a document i is used to determine the relative position during splicing of image tile, and reference number of a document i is by image tile file line, row number
Divided by 2 after remainder, the two-dimensional coordinate (R%2, C%2) of acquisition is calculated by formula (2 × (R%2)+C%2) and converts and obtains；
Wherein R represents image tile line number, and C represents image tile row number；
Step 402：The image tile data of identical key value Key can be sent to same reduction function (hereinafter referred
Reduce functions), Reduce functions first arrange the tile file ascending order in List lists according to i values, after then sorting
File according to lower-left, upper left, bottom right, bottom right position relationship carry out splicing merger, obtain a new shadow named with key assignments
As tile file；New one layer of image tile file is all after generation, and new image tile layer data is used as MapReduce's again
Input data generates more last layer image tile data, terminates until pyramid model is built.
Operation principle：The extensive image data of input is first grouped by the present invention, then can will be directed to certain layer
Extraction task be assigned to by way of Map on each back end perform, system has data and meter in this process
The action of migration is calculated, calls Reduce functions that each back end is calculated merger respectively after the completion of back end processing, most
The image of the small one-level of resolution ratio is obtained eventually, necessary layer is extracted with the order from bottom to upper strata successively, until obtaining whole gold
Untill word tower model has been built.
Beneficial effect：Compared with prior art, the present invention is with good expansibility, the pyramid parallel constructing method
Based on MapReduce parallel computation patterns, easily hardware infrastructure can be operationally adjusted, be added deduct as increased
Few number of nodes, is adjusted to hard disk.Meanwhile, it is parallel on computer cluster to build pyramid model, for magnanimity number
According to higher efficiency；Centralization builds image pyramid and is only limited to the low volume data of small range and handles slow in the case of unit
Slowly, even disk size can accommodate initial data, for the treatment effeciency ten of the extensive image such as multidate, multi-data source
Divide low.Under cluster environment, upper strata is concurrently generated according to pyramid bottom tile based on the independence between tile by the present invention
Tile, makes full use of the storage resource and calculating advantage of cluster, especially for the big region (such as global range) of construction, big rule
The tile pyramid model of modulus evidence is advantageously.
Brief description of the drawings
Fig. 1 is flow chart of the invention；
Fig. 2 is the structural representation of tile pyramid model.
Embodiment
Technical solution of the present invention is described in detail below, but protection scope of the present invention is not limited to the implementation
Example.
Embodiment：The MapReduce used in this patent is that Google was applied to large-scale cluster in proposition in 2004
The parallel computational model of data processing is carried out, is also the core calculations pattern of current cloud computing.
As shown in figure 1, a kind of tile pyramid parallel constructing method based on MapReduce, comprises the following steps：
Step 10：Pyramid bottom is used as using ultimate resolution image；
Step 20：Piecemeal is carried out to bottom, and the image blocks being divided into are encoded；Wherein, using the cutting of quaternary tree
Mode carries out piecemeal to image.The image tile being blocked into is encoded by the way of based on grid and Hilbert codings；
The numbering of each image tile is by zoom level Z, Hilbert is encoded, tile row number and tile line number are constituted.
Step 30：Image blocks data input after block encoding in step 20 is generated new one into MapReduce model
The tile data of layer；
Step 40：New one layer of the tile data that MapReduce model is often taken turns to generation successively is used as next round
The input data of MapReduce model, generates more last layer tile data, terminates until pyramid model is built.
As shown in Fig. 2 the pyramidal generation strategy of tile is：First initial image is carried out as the N layers of pyramid model
2N×2NPiecemeal after just obtained the tile matrix of bottom, then taken based on this according to every 2 × 2 pixels and be worth to 1
The rule of individual pixel obtains N-1 layers, then carries out 2N-1×2N-1N-1 layers of tile are obtained after piecemeal, by that analogy, in preceding layer
On the basis of generate later layer, untill the 0th layer of tile is ultimately generated.
From the pyramidal vertical mapping relations built between principle, pyramid levels of tile, four shadows of lower floor
If as block by hilbert encode divided by 4 round, ranks number respectively divided by 2 round and can obtain father's image block coding on upper strata,
Based on this mapping relations, the parallel structure of pyramid model can be realized based on MapReduce.
As shown in table 1, wherein Z represents zoom level, H and represents Hilbert volumes Map and Reduce input/output argument
Code, C represent tile row number, R and represent tile line number, and Z_H_C_R.dat represents corresponding image tile file.
Table 1
The parallel algorithm for building pyramid model is as follows：
1st, the Map stages
For each image tile file Z_H_C_R.dat of this layer, a series of key-value pairs are produced by Map functionsWherein key is last layer father's image tiles code, by this layer of tile text
The zoom level of part subtracts 1, No. Hilbert divided by 4 rounds, ranks number divided by 2 round acquisition；Value is by reference number of a document i and tile
File is constituted, and wherein i is used to determine relative position during tile splicing, and its principle is four tile files of father's block number identical
Value after ranks number divided by 2 remainders is (0,0), (0,1), (1,0), (1,1), mapped respectively during splicing lower-left, upper left, bottom right,
The position of bottom right, here for the convenience of sequence, this two-dimensional position relation is calculated by (2 × (R%2)+C%2) and converted
For the reference number of a document i of string relation.
2nd, the Reduce stages
The tile data of identical key assignments can be sent to same Reduce functions, and Reduce functions first will according to i values
In List lists tile file ascending order arrangement, then by the file after sequence according to lower-left, upper left, bottom right, bottom right position
Relation carries out splicing merger, obtains a new tile file named with key assignments.After new one layer of tile file is all generated, newly
Tile layer data again as MapReduce input data generate more last layer tile data, until pyramid model build
Terminate.
By taking the N level tile pyramid models for generating certain Raster Images data as an example, illustrate the implementation process of the present invention：
1. cloud computing environment is built., can be using physical machine as calculate node exemplified by using Hadoop as cloud platform, or lead to
Cross and virtual machine is installed in physical machine to increase calculate node number, be these node distributions role (Master nodes or Worker
Node) and JDK is installed, the compiling and MapReduce operation that are Hadoop are ready；OpenSSH is installed, configuration SSH exempts from
Password login.JDK and SSH are installed after configuration well, and Hadoop is installed and configured as cloud computing platform, and to related configuration
File is set.
2. piecemeal processing is carried out to raw video data, raw video data can be made up of several images, to raw video
The cutting size of data typically takes 2 integer power, generally using 256 × 256 or 512 × 512, if the pixel of tile be M ×
M, then must assure that the resolution ratio of raw video data meets M × M × 2N×2N。
3. in raw video data are carried out with piecemeal processing procedure, the scope according to representated by tile calls tiles code
Algorithm is named for each tile.
4. the image data of bottom after piecemeal as MapReduce input file, call based on MapReduce's and
Row tile pyramid construction algorithm, untill the 0th grade of tile is generated.
So far, the pyramidal construction work of tile has been completed.
Claims (1)
1. a kind of tile pyramid parallel constructing method based on MapReduce, it is characterised in that：Comprise the following steps：
Step 10：Pyramid bottom is used as using ultimate resolution image；
Step 20：Piecemeal is carried out to bottom, and the image blocks being divided into are encoded；
Step 30：Image blocks data input after block encoding in step 20 is generated new one layer into MapReduce model
Tile data；
Step 40：New one layer of the tile data that MapReduce model is often taken turns to generation successively is used as next round MapReduce moulds
The input data of type, generates more last layer tile data, terminates until pyramid model is built；
Method of partition in the step 20 is：The slit mode of quaternary tree carries out piecemeal to image；
Coding method in the step 20 is：Image watt by the way of based on grid and Hilbert codings to being blocked into
Piece is encoded；The numbering of each image tile is by zoom level Z, Hilbert coding, tile row number and tile line number group
Into；
The method of every layer of tile data generation is：
Step 401：Each image tile file Z_H_C_R.dat of place layer produces a series of key-value pair Key by Map functions
->Value, wherein key are last layer father's image tiles code, subtract 1 by the zoom level of this floor tile file, No. Hilbert
Divided by 4 round, row, column number respectively divided by 2 round acquisition；Value is made up of reference number of a document i and tile file；Wherein file is compiled
Number i is used to determine the relative position during splicing of image tile, and reference number of a document i is by image tile file line, row number divided by 2 remainders
Afterwards, by the two-dimensional coordinate of acquisition（R%2, C%2）Pass through formula（2×（R%2）+ C%2）Calculate conversion and obtain；Wherein R represents image
Tile line number, C represents image tile row number；
Step 402：The image tile data of identical key value Key can be sent to same Reduce functions, and Reduce functions are first
First according to i values by List lists tile file ascending order arrangement, then by the file after sequence according to lower-left, upper left, bottom right,
The position relationship of bottom right carries out splicing merger, obtains a new image tile file named with key assignments；New one layer of image watt
Piece file is all after generation, and new image tile layer data generates more last layer image as MapReduce input data again
Tile data, terminates until pyramid model is built.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410305679.5A CN104091301B (en) | 2014-06-30 | 2014-06-30 | A kind of tile pyramid parallel constructing method based on MapReduce |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410305679.5A CN104091301B (en) | 2014-06-30 | 2014-06-30 | A kind of tile pyramid parallel constructing method based on MapReduce |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104091301A CN104091301A (en) | 2014-10-08 |
| CN104091301B true CN104091301B (en) | 2017-08-25 |
Family
ID=51639016
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410305679.5A Expired - Fee Related CN104091301B (en) | 2014-06-30 | 2014-06-30 | A kind of tile pyramid parallel constructing method based on MapReduce |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104091301B (en) |
Families Citing this family (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104657436B (en) * | 2015-02-02 | 2019-01-25 | 中国人民解放军空军航空大学 | Static tile pyramid parallel constructing method based on MapReduce |
| WO2016197223A1 (en) * | 2015-06-10 | 2016-12-15 | Faculdades Católicas, Mantenedora Da Pontifícia Universidade Católica Do Rio De Janeiro - Puc-Rio | Method for assisting digital vectorial and raster data processing in a computer cluster |
| CN105608222A (en) * | 2016-01-12 | 2016-05-25 | 中国人民解放军国防科学技术大学 | Rapid building method of tile pyramid for large-scale raster data set |
| CN106251374B (en) * | 2016-07-21 | 2019-05-10 | 深圳市检验检疫科学研究院 | MapReduce data processing method in Hadoop based on Zigzag |
| CN106528699A (en) * | 2016-11-02 | 2017-03-22 | 北京航天泰坦科技股份有限公司 | IO multi-thread computing method and system for fast building image pyramid |
| CN106991135B (en) * | 2017-03-15 | 2020-07-24 | 江苏物联网研究发展中心 | Rapid tile generation method for remote sensing image data |
| CN106991143B (en) * | 2017-03-22 | 2019-07-19 | 苏州中科图新网络科技有限公司 | Multi-layer image file, generation method and device, read method and device |
| CN107679164A (en) * | 2017-09-28 | 2018-02-09 | 上海交通大学 | The browsing method and system of the vast capacity image virtually shown based on quaternary tree |
| CN110109751B (en) * | 2019-04-03 | 2022-04-05 | 百度在线网络技术（北京）有限公司 | Distribution method and device of distributed graph cutting tasks and distributed graph cutting system |
| CN111061806B (en) * | 2019-11-21 | 2023-04-07 | 中国航空无线电电子研究所 | Storage method and networked access method for distributed massive geographic tiles |
| CN111552753B (en) * | 2020-04-24 | 2020-12-29 | 中国科学院空天信息创新研究院 | Massive remote sensing data organization and management method and system based on distributed hbase storage |
| CN113157654B (en) * | 2021-05-12 | 2022-12-02 | 山东志盈医学科技有限公司 | Method and apparatus for digital slice image storage |
| CN115470366A (en) * | 2022-08-31 | 2022-12-13 | 湖南省第二测绘院 | Tile-based remote sensing image storage method and system |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103281376B (en) * | 2013-05-31 | 2015-11-11 | 武汉大学 | The automatic buffer memory construction method of magnanimity sequential remote sensing image under a kind of cloud environment |
| CN103455624B (en) * | 2013-09-16 | 2017-07-25 | 湖北文理学院 | A kind of global multidimensional remote sensing image Map Service of Network implementation method of lightweight |
-
2014
- 2014-06-30 CN CN201410305679.5A patent/CN104091301B/en not_active Expired - Fee Related
Also Published As
| Publication number | Publication date |
|---|---|
| CN104091301A (en) | 2014-10-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104091301B (en) | A kind of tile pyramid parallel constructing method based on MapReduce | |
| JP6009075B2 (en) | Particle flow simulation system and method | |
| CN110599490B (en) | Remote sensing image data storage method and system | |
| Li et al. | Performance improvement techniques for geospatial web services in a cyberinfrastructure environment–A case study with a disaster management portal | |
| CN105786942B (en) | A kind of geography information storage system based on cloud platform | |
| CN105446896B (en) | The buffer memory management method and device of MapReduce application | |
| CN110321443B (en) | Three-dimensional live-action model database construction method and device and data service system | |
| CN103403769A (en) | Planetary scale object rendering | |
| CN101661631B (en) | Point model rapid lossless compression processing method based on local minimum spanning tree | |
| US20140278228A1 (en) | Mapping uncertain geometries to graticules | |
| CN109936745A (en) | For improving the method and system of the decompression of original video data | |
| CN102255873B (en) | Method for transmission of vector data on the basis of pixel non-destructive compression of ordered point set | |
| CN105045856A (en) | Hadoop-based data processing system for big-data remote sensing satellite | |
| US20220207370A1 (en) | Inferring device, training device, inferring method, and training method | |
| CN102693306A (en) | On-line linkage update service method of multi-scale geographic data | |
| CN104570081A (en) | Pre-stack reverse time migration seismic data processing method and system by integral method | |
| US20210182357A1 (en) | System and method for model parameter optimization | |
| WO2022147923A1 (en) | Block network safety organization storage mapping method for spatial data | |
| Guo et al. | An efficient data organization and scheduling strategy for accelerating large vector data rendering | |
| CN111061806B (en) | Storage method and networked access method for distributed massive geographic tiles | |
| CN108520532B (en) | Method and device for identifying motion direction of object in video | |
| KR20200065797A (en) | 3D spatial information visualization system and method | |
| CN110765130B (en) | Ripley's K function-based spatio-temporal POI data point pattern analysis method in distributed environment | |
| CN116128701A (en) | Device and method for executing graph calculation task | |
| CN111522899B (en) | Parallel compression method and device for high compression ratio of three-dimensional vector data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C41 | Transfer of patent application or patent right or utility model | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20170215 Address after: 102206 Beijing city Changping District Zhu Daxinzhuang North Agricultural Road No. 2 Applicant after: North China Electric Power University Address before: Shishi Fuda Road Park 212000 city of Jiangsu Province, Zhenjiang Jurong Economic Development Zone, building 02, room 315 Applicant before: Jiangsu Huada Tianyi Electric Power Science & Technology Co., Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20170825 Termination date: 20180630 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |