CN103809969A - Remote-sensing image data parallel resampling method based on pre-fragmentation in cloud environment - Google Patents
Remote-sensing image data parallel resampling method based on pre-fragmentation in cloud environment Download PDFInfo
- Publication number
- CN103809969A CN103809969A CN201410017598.5A CN201410017598A CN103809969A CN 103809969 A CN103809969 A CN 103809969A CN 201410017598 A CN201410017598 A CN 201410017598A CN 103809969 A CN103809969 A CN 103809969A
- Authority
- CN
- China
- Prior art keywords
- image
- image data
- burst
- class
- resampling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Abstract
The invention discloses a remote-sensing image data parallel resampling method based on pre-fragmentation in a cloud environment. The method includes the following steps that firstly, target image data are uploaded to a cloud platform HDFS; secondly, pre-fragmentation processing is carried out on the image data so that the image data can be partitioned according to regular grids; thirdly, each image block obtained after pre-fragmentation serves as one independent image fragment; fourthly, a map parallel task is started for each image fragment, parallel processing is carried out on different group nodes, and the original image fragments are resampled layer by layer; fifthly, the image fragments are merged according to image hierarchies, and splicing and merging are carried out on image fragment sets on each hierarchy; sixthly, merged image data on each resolution ratio hierarchy are output to the HDFS. The method is suitable for resampling of image data, is also suitable for parallel processing of image data capable of being divided from other data, and is important for remote-sensing data cloud computing services.
Description
Technical field
The present invention relates to cloud computing, Geographic Information System and remote sensing image data process field, relate in particular to the parallel method for resampling of the remote sensing image data based on pre-burst under a kind of cloud environment.
Background technology
For large-scale high-definition remote sensing image data is carried out to hierarchical block tissue, foundation has the image pyramid of multiresolution level, make the directly image data piece of request target resolution level, target area of client, raw video data must be passed through resampling, thereby obtain the image data of a low class resolution ratio.
In image data processing procedure, although many operations are finally that certain pixel is processed, in processing procedure, conventionally need to consider the pixel of image adjacency.But under cloud computing environment, MapReduce parallel computation framework uses stream data reading manner, under default situations, data are carried out to mode that burst reads as shown in Figure 2, this mode has been ignored original spatial relationship between pixel, is difficult to the remote sensing image process field in order to need to consider adjacent pixels.
Summary of the invention
The object of the invention is to overcome the deficiencies in the prior art, the parallel method for resampling of the remote sensing image data based on pre-burst under a kind of cloud environment is provided.
The parallel method for resampling of remote sensing image data based on pre-burst under cloud environment comprises the steps:
(1) target image data upload
Before parallel resampling, client, by calling the Java API of Hadoop, writes pending remote sensing image data on the distributed file system HDFS cluster of cloud storage, to process by MapReduce parallel computation framework;
(2) the pre-burst of image data
For the data input policing that is applicable to image data parallel computation is provided, make image data in the time of burst, can keep original spatial relation between pixel, the target image data that are uploaded on HDFS cluster in step (1) are carried out to piecemeal according to regular grid mode, and these piecemeals are write back on HDFS cluster;
(3) image data input
By the pre-burst image blocks generating in MapReduce framework read step (2), using each image blocks as one independently burst ImageSplit input, layer level number and the ranks position of recording image piece, wherein the layer level number of raw video is 0, simultaneously, forbid that MapReduce framework further carries out again piecemeal and member record to it, thereby each image blocks is processed as an independent image burst;
(4) parallel resampling
Be that each image burst ImageSplite in step (3) creates independent resampling map task by MapReduce framework, these map tasks in parallel operate on the each node of cluster, respectively corresponding image burst is carried out to resampling processing, generate various level image burst, and carried out unique identification with floor level number and ranks number;
(5) image data merger
The image data that is each level by MapReduce framework creates an independent reduce task, these reduce tasks in parallel operate on the each node of cluster, the reduce task that the image burst that has identical layer level number is assigned to respective layer level number is carried out merger processing, and the each layer image burst after merger identifies by layer level number;
(6) image data output stage
According to different resolution layer level number, the image data after merger in step (5) is successively exported according to object format, and it is upper to write distributed file system HDFS, thereby completes the result output of the parallel resampling of image data;
Described step (3) comprises design ImageFileInputFormat class and ImageRecordReader class; ImageFileInputFormat class is inherited the class in FileInputFormat, controls the image blocks through pre-burst no longer continue burst by such, and each image blocks is inputted as an independent image burst; ImageRecordReader class, inherits the class in RecordReader, realizes each image burst process as an independent record by such; Design ImageBlockWritable class realizes the encapsulation to image burst, has comprised image blocks binary stream, ranks position and layer level number, just to process by MapReduce program.
ResaMapper class that described step (4) has been included as resampling task definition, inherits the class in Mapper, in map () method, load raw video burst, i.e. the 0th grade of image burst, and the resampling that circulates, generates each level image data burst; After carrying out each level resampling, to image burst according to <IntWritable, ImageBlockWritable> key-value pair mode is exported, continue to process for next step, wherein IntWritable is layer level number, and ImageBlockWritable is serializability image data piece.
Described step (5) is included as the merger design ResaReducer class of image burst, inherits the class in Reducer, for realizing image data merger reduce task; In the time carrying out merger processing, the image burst of same level is assigned to same ResaReducer according to layer level number and goes out, and completes the splicing of all image bursts of place resolution level by reduce () method; The image merger reduce task of different resolution level is assigned on different nodes to be carried out, thus the spliced image data of the each level of parallel generation, and identify by layer level number.
Described step (6) is included as the output design ImageOutputFormat class and the ImageRecordWriter class that realize image data, ImageOutputFormat class is inherited the class in FileOutputFormat, be used for controlling data output format, ImageRecordWriter class is inherited the class in RecordWriter, realize each layer image <IntWritable by write () method, the file file output of BytesWritable > key-value pair, wherein IntWritable is layer level number, BytesWritable is the spliced image data binary stream of this level.
The beneficial effect that the present invention compared with prior art has:
(1) realize the parallel resampling of remote sensing image data under cloud computing environment, compared with traditional method for resampling, greatly improved the efficiency of resampling;
(2) can, by the high-performance calculation ability of cloud computing for the parallel processing of remote sensing image data, promote the combination of cloud computing technology and remote sensing image data Storage and Processing technology, for remotely-sensed data cloud service provides technical foundation.
Accompanying drawing explanation
Fig. 1 is that in MapReduce of the present invention, image divides sheet mode in advance;
Fig. 2 is image acquiescence point sheet mode in MapReduce of the present invention;
Fig. 3 is image blocks serializability class of the present invention (ImageBlockWritable) definition.
Embodiment
The parallel method for resampling of remote sensing image data based on pre-burst under cloud environment comprises the steps:
(1) target image data upload
Before parallel resampling, client, by calling the Java API of Hadoop, writes pending remote sensing image data on the distributed file system HDFS cluster of cloud storage, to process by MapReduce parallel computation framework;
(2) the pre-burst of image data
For the data input policing that is applicable to image data parallel computation is provided, make image data in the time of burst, can keep original spatial relation between pixel, the target image data that are uploaded on HDFS cluster in step (1) are carried out to piecemeal according to regular grid mode, and these piecemeals are write back on HDFS cluster;
(3) image data input
By the pre-burst image blocks generating in MapReduce framework read step (2), using each image blocks as one independently burst ImageSplit input, layer level number and the ranks position of recording image piece, wherein the layer level number of raw video is 0, simultaneously, forbid that MapReduce framework further carries out again piecemeal and member record to it, thereby each image blocks is processed as an independent image burst;
(4) parallel resampling
Be that each image burst ImageSplite in step (3) creates independent resampling map task by MapReduce framework, these map tasks in parallel operate on the each node of cluster, respectively corresponding image burst is carried out to resampling processing, generate various level image burst, and carried out unique identification with floor level number and ranks number;
(5) image data merger
The image data that is each level by MapReduce framework creates an independent reduce task, these reduce tasks in parallel operate on the each node of cluster, the reduce task that the image burst that has identical layer level number is assigned to respective layer level number is carried out merger processing, and the each layer image burst after merger identifies by layer level number;
(6) image data output stage
According to different resolution layer level number, the image data after merger in step (5) is successively exported according to object format, and it is upper to write distributed file system HDFS, thereby completes the result output of the parallel resampling of image data;
Described step (3) comprises design ImageFileInputFormat class and ImageRecordReader class; ImageFileInputFormat class is inherited the class in FileInputFormat, controls the image blocks through pre-burst no longer continue burst by such, and each image blocks is inputted as an independent image burst; ImageRecordReader class, inherits the class in RecordReader, realizes each image burst process as an independent record by such; Design ImageBlockWritable class realizes the encapsulation to image burst, has comprised image blocks binary stream, ranks position and layer level number, just to process by MapReduce program.
ResaMapper class that described step (4) has been included as resampling task definition, inherits the class in Mapper, in map () method, load raw video burst, i.e. the 0th grade of image burst, and the resampling that circulates, generates each level image data burst; After carrying out each level resampling, to image burst according to <IntWritable, ImageBlockWritable> key-value pair mode is exported, continue to process for next step, wherein IntWritable is layer level number, and ImageBlockWritable is serializability image data piece.
Described step (5) is included as the merger design ResaReducer class of image burst, inherits the class in Reducer, for realizing image data merger reduce task; In the time carrying out merger processing, the image burst of same level is assigned to same ResaReducer according to layer level number and goes out, and completes the splicing of all image bursts of place resolution level by reduce () method; The image merger reduce task of different resolution level is assigned on different nodes to be carried out, thus the spliced image data of the each level of parallel generation, and identify by layer level number.
Described step (6) is included as the output design ImageOutputFormat class and the ImageRecordWriter class that realize image data, ImageOutputFormat class is inherited the class in FileOutputFormat, be used for controlling data output format, ImageRecordWriter class is inherited the class in RecordWriter, realize each layer image <IntWritable by write () method, the file file output of BytesWritable> key-value pair, wherein IntWritable is layer level number, BytesWritable is the spliced image data binary stream of this level.
Embodiment:
The present embodiment is realized on cloud computing architecture Hadoop increasing income, and implementation step is as follows:
(1) target image data upload
The parallel resampling of remote sensing image data under cloud environment is under cluster environment, application MapReduce parallel computation framework carries out parallel processing to image data, therefore before resampling, user need to by target image data upload on the HDFS cluster of cloud platform further to process.
(2) the pre-burst of image data
This step is the data input policing for a kind of applicable image data parallel computation is provided, and makes image data in the time of burst, can keep its original spatial relation.As shown in Figure 1, the target image data on HDFS cluster are carried out to suitably piecemeal according to regular grid, and it is upper to write back HDFS, further processes for MapReduce Parallel Computation.
(3) image data input
In this step, MapReduce framework reads the pre-burst image blocks generating in previous step, and using each image blocks as one independently burst (ImageSplit) input, forbid that MapReduce framework further carries out again piecemeal and member record to it simultaneously.Thereby each image blocks is processed as an independent image burst.
Because the processing of image normally needs to consider neighbour's pixel, this step has been avoided the destruction of position relationship between image pixel that MapReduce acquiescence image data partitioned mode (as shown in Figure 2) causes.
For supporting the input of image blocks, design ImageFileInputFormat class (inheriting the class in FileInputFormat) and ImageRecordReader class (inheriting the class in RecordReader).Wherein, ImageFileInputFormat class, for controlling the burst of image, is controlled the image blocks through pre-burst by such and is no longer continued burst.In the time of the reading of image blocks, the filename form of image blocks should adopt ranks position readme form, it is the position that each filename itself can show this image blocks, as " row-column " form, after reading, resolved by ImageRecordReader, and write ImageBlockWritable.Wherein ImageBlockWritable class is the image data piece class of the serializability of this patent design, and it defines as shown in Figure 3.
(4) the parallel resampling stage
In this step, MapReduce framework is a corresponding map task of the each establishment of each image burst ImageSplite, these map tasks in parallel operate on the each node of cluster, respectively corresponding image burst is carried out to resampling processing, this memory node data are carried out to priority processing simultaneously, replace Mobile data thereby realize with mobile computing.
For realizing above-mentioned functions, the sampling task of need to attaching most importance to definition ResaMapper class (inheriting the class in Mapper), in the map of ResaMapper class () method, has loaded raw video data block (the 0th grade), to the resampling that circulates of each level.After each level resampling, to image blocks, according to <IntWritable, ImageBlockWritable > key-value pair mode is exported.Wherein IntWritable is layer level number, and ImageBlockWritable is serializability image data piece.
(5) image data merging phase
In the merging phase of image data, MapReduce framework merges the image data piece of same level by reduce () method.This patent design ResaReducer class (inheriting the class in Reducer) is carried out merger (realizing the splicing processing of image blocks) to image blocks ImageBlockWritable.
In this step, the image blocks of same level being assigned to same ResaReducer according to layer level number merges.Due to multiple ResaReducer different node executed in parallel on cluster, therefore also can realize as required the parallel warehouse-in of image data in this stage.
(6) image data output stage
This stage control output format, and to the image <IntWritable after each layer of merger, BytesWritable> key-value pair is exported, wherein IntWritable is layer level number, and BytesWritable is this layer spliced image data binary stream again.This patent design ImageOutputFormat class (inheriting the class in FileOutputFormat) and ImageRecordWriter class (inheriting the class in RecordWriter) realize the output of image data.
Can realize the remote sensing image efficient parallel resampling based on MapReduce by above step.Meanwhile, this treatment scheme is not only applicable to the resampling of image data, is applicable to the image data parallel processing field that other data can be divided yet, and is the technical foundation of remotely-sensed data cloud computing service.
Claims (5)
1. the parallel method for resampling of the remote sensing image data based on pre-burst under cloud environment, is characterized in that comprising the steps:
(1) target image data upload
Before parallel resampling, client, by calling the Java API of Hadoop, writes pending remote sensing image data on the distributed file system HDFS cluster of cloud storage, to process by MapReduce parallel computation framework;
(2) the pre-burst of image data
For the data input policing that is applicable to image data parallel computation is provided, make image data in the time of burst, can keep original spatial relation between pixel, the target image data that are uploaded on HDFS cluster in step (1) are carried out to piecemeal according to regular grid mode, and these piecemeals are write back on HDFS cluster;
(3) image data input
By the pre-burst image blocks generating in MapReduce framework read step (2), using each image blocks as one independently burst ImageSplit input, layer level number and the ranks position of recording image piece, wherein the layer level number of raw video is 0, simultaneously, forbid that MapReduce framework further carries out again piecemeal and member record to it, thereby each image blocks is processed as an independent image burst;
(4) parallel resampling
Be that each image burst ImageSplite in step (3) creates independent resampling map task by MapReduce framework, these map tasks in parallel operate on the each node of cluster, respectively corresponding image burst is carried out to resampling processing, generate various level image burst, and carried out unique identification with floor level number and ranks number;
(5) image data merger
The image data that is each level by MapReduce framework creates an independent reduce task, these reduce tasks in parallel operate on the each node of cluster, the reduce task that the image burst that has identical layer level number is assigned to respective layer level number is carried out merger processing, and the each layer image burst after merger identifies by layer level number;
(6) image data output stage
According to different resolution layer level number, the image data after merger in step (5) is successively exported according to object format, and it is upper to write distributed file system HDFS, thereby completes the result output of the parallel resampling of image data.
2. method according to claim 1, is characterized in that described step (3) comprises design ImageFileInputFormat class and ImageRecordReader class; ImageFileInputFormat class is inherited the class in FileInputFormat, controls the image blocks through pre-burst no longer continue burst by such, and each image blocks is inputted as an independent image burst; ImageRecordReader class, inherits the class in RecordReader, realizes each image burst process as an independent record by such; Design ImageBlockWritable class realizes the encapsulation to image burst, has comprised image blocks binary stream, ranks position and layer level number, just to process by MapReduce program.
3. method according to claim 1, ResaMapper class that step (4) described in it is characterized in that has been included as resampling task definition, inherit the class in Mapper, in map () method, load raw video burst, i.e. the 0th grade of image burst, and the resampling that circulates, generates each level image data burst; After carrying out each level resampling, to image burst according to <IntWritable, ImageBlockWritable> key-value pair mode is exported, continue to process for next step, wherein IntWritable is layer level number, and ImageBlockWritable is serializability image data piece.
4. method according to claim 1, is characterized in that described step (5) is included as the merger design ResaReducer class of image burst, inherits the class in Reducer, for realizing image data merger reduce task; In the time carrying out merger processing, the image burst of same level is assigned to same ResaReducer according to layer level number and goes out, and completes the splicing of all image bursts of place resolution level by reduce () method; The image merger reduce task of different resolution level is assigned on different nodes to be carried out, thus the spliced image data of the each level of parallel generation, and identify by layer level number.
5. method according to claim 1, it is characterized in that described step (6) is included as the output design ImageOutputFormat class and the ImageRecordWriter class that realize image data, ImageOutputFormat class is inherited the class in FileOutputFormat, be used for controlling data output format, ImageRecordWriter class is inherited the class in RecordWriter, realize each layer image <IntWritable by write () method, the file file output of BytesWritable > key-value pair, wherein IntWritable is layer level number, BytesWritable is the spliced image data binary stream of this level.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410017598.5A CN103809969B (en) | 2014-01-15 | The parallel method for resampling of remote sensing image data based on pre-burst under cloud environment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410017598.5A CN103809969B (en) | 2014-01-15 | The parallel method for resampling of remote sensing image data based on pre-burst under cloud environment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103809969A true CN103809969A (en) | 2014-05-21 |
| CN103809969B CN103809969B (en) | 2016-11-30 |
Family
ID=
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104537682A (en) * | 2015-01-26 | 2015-04-22 | 四川众亿得科技有限责任公司 | Medical image segmenting and dispatching method |
| CN105893620A (en) * | 2016-04-28 | 2016-08-24 | 江苏物联网研究发展中心 | Massive super-large image accessing method based on distributed database |
| CN105930794A (en) * | 2016-04-20 | 2016-09-07 | 东北大学 | Indoor scene identification method based on cloud computing |
| CN106503093A (en) * | 2016-10-13 | 2017-03-15 | 华东师范大学 | A kind of large-scale remote sensing image reconstructing method suitable for DCE |
| CN108090865A (en) * | 2017-12-15 | 2018-05-29 | 武汉大学 | The in-orbit real-time streaming processing method of optical satellite remote sensing image and system |
| CN110855638A (en) * | 2019-10-28 | 2020-02-28 | 中国资源卫星应用中心 | Remote sensing satellite data decompression processing system and method based on cloud computing |
Non-Patent Citations (2)
| Title |
|---|
| 刘义 等: "利用MapReduce进行批量遥感影像瓦片金字塔构建", 《武汉大学学报.信息科学版》 * |
| 刘磊 等: "基于开源Hadoop的栅格数据分布式处理", 《华中科技大学学报（自然科学版）》 * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104537682A (en) * | 2015-01-26 | 2015-04-22 | 四川众亿得科技有限责任公司 | Medical image segmenting and dispatching method |
| CN104537682B (en) * | 2015-01-26 | 2018-09-07 | 四川众亿得科技有限责任公司 | A kind of segmentation of medical image and dispatching method |
| CN105930794A (en) * | 2016-04-20 | 2016-09-07 | 东北大学 | Indoor scene identification method based on cloud computing |
| CN105893620A (en) * | 2016-04-28 | 2016-08-24 | 江苏物联网研究发展中心 | Massive super-large image accessing method based on distributed database |
| CN106503093A (en) * | 2016-10-13 | 2017-03-15 | 华东师范大学 | A kind of large-scale remote sensing image reconstructing method suitable for DCE |
| CN106503093B (en) * | 2016-10-13 | 2019-06-11 | 华东师范大学 | A kind of large-scale remote sensing image reconstructing method suitable for distributed computing environment |
| CN108090865A (en) * | 2017-12-15 | 2018-05-29 | 武汉大学 | The in-orbit real-time streaming processing method of optical satellite remote sensing image and system |
| CN108090865B (en) * | 2017-12-15 | 2021-06-15 | 武汉大学 | Optical satellite remote sensing image on-orbit real-time streaming processing method and system |
| CN110855638A (en) * | 2019-10-28 | 2020-02-28 | 中国资源卫星应用中心 | Remote sensing satellite data decompression processing system and method based on cloud computing |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10085056B2 (en) | Method and system for improving application sharing by dynamic partitioning | |
| TWI696079B (en) | Multi-blockchain network data processing method, device and server | |
| Wiley et al. | Astronomy in the cloud: using mapreduce for image co-addition | |
| US8410993B2 (en) | Systems, methods, and devices for highly interactive large image display and manipulation on tiled displays | |
| US8605081B2 (en) | Converting 3D data to hogel data | |
| AU2018215981A1 (en) | Apparatus and method for providing a virtual device | |
| CN104091301B (en) | A kind of tile pyramid parallel constructing method based on MapReduce | |
| JP6636647B2 (en) | Method and apparatus for generating a random character string | |
| TWI694700B (en) | Data processing method and device, user terminal | |
| CN111767144B (en) | Transaction route determination method, device, equipment and system for transaction data | |
| CN105867944B (en) | Web front end data operation layer and its implementation | |
| CN104331545A (en) | Realization method of power network automation GIS (Geographic Information System) electrical layer based on CIM/G | |
| TW202008762A (en) | Data processing method and device, client, and server | |
| CN106557307A (en) | The processing method and processing system of business datum | |
| CN104660690A (en) | Cloud video service monitoring system | |
| CN103559247A (en) | Data service processing method and device | |
| KR20200084695A (en) | Apparatus and method for managing application | |
| CN103809969A (en) | Remote-sensing image data parallel resampling method based on pre-fragmentation in cloud environment | |
| CN103809969B (en) | The parallel method for resampling of remote sensing image data based on pre-burst under cloud environment | |
| CN102984571A (en) | External data reading method of Gstreamer in digital television and device thereof | |
| CN105045726A (en) | Picture operation method based on parallel computation and picture operation system based on parallel computation | |
| CN105306858A (en) | Video data storage method and device | |
| CN111190607B (en) | Task plugin processing method and device, task scheduling server and storage medium | |
| CN112486679A (en) | Pod scheduling method, device and equipment for kubernets cluster | |
| CN113311995B (en) | Cloud hard disk preparation method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |