CN103903272B - A kind of StaMPS Algorithm parallelization processing method based on Hadoop - Google Patents
A kind of StaMPS Algorithm parallelization processing method based on Hadoop Download PDFInfo
- Publication number
- CN103903272B CN103903272B CN201410152271.9A CN201410152271A CN103903272B CN 103903272 B CN103903272 B CN 103903272B CN 201410152271 A CN201410152271 A CN 201410152271A CN 103903272 B CN103903272 B CN 103903272B
- Authority
- CN
- China
- Prior art keywords
- image
- hadoop
- registration
- point
- space
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Landscapes
- Image Processing (AREA)
Abstract
A kind of StaMPS Algorithm parallelization processing method based on Hadoop, described Hadoop includes HDFS and MapReduce, including: essence registration, run in Hadoop cloud platform, it is achieved the registration of image, registration unit serial process is transform as multi-host parallel；PS analyzes, and runs in Hadoop cloud platform, it is achieved the selection of PS point, simplify, and the process of unit serial order is transform as multi-host parallel.Described essence registration includes: the essence registration of auxiliary image to main image and the essence registration of auxiliary image to auxiliary image.In the present invention, essence registration is achieved parallel by serial transformation, the PS analysis phase achieves parallel from step 1 to step 7, combination by Hadoop and StaMPS algorithm, StaMPS algorithm process efficiency is increased exponentially, adapt to storage and the calculating of the mass data of big data age, improve availability and the autgmentability of system.
Description
Technical field
The present invention relates to Hadoop technology, particularly relate to a kind of StaMPS Algorithm parallelization processing method based on Hadoop.
Background technology
Along with various remote sensing instrument spatial resolutions, the improving constantly of spectral resolution, prolongation over time, the data volume sharp increase of remote sensing image, have accumulated substantial amounts of image data.In the face of the growth of data explosion formula, conventional individual serial process mode is the most slowly fallen behind, and emerging distributed variable-frequencypump is increasingly mature, will substitute original processing mode.At present the magnanimity about data processes and stores becomes the upsurge of research.Among these, by Doug
The Hadoop platform of Cutting et al. exploitation is shown one's talent, and becomes the favorite of distributed treatment.Hadoop distributed structure/architecture is mainly made up of HDFS distributed file system and MapReduce computation module.HDFS is mainly responsible for storage mass data, and MapReduce is mainly responsible for calculating mass data.And traditional StaMPS algorithm process remotely-sensed data the most all uses shell and the matlab script serial process of standalone, in mass data processing today, some is unable to do what one wishes, hence in so that StaMPS algorithm process mass remote sensing data and Hadoop are combined into possibility.
How Hadoop is applied to StaMPS algorithm Ground Subsidence Monitoring, it is achieved the storage of mass remote sensing data and calculating, improves efficiency as much as possible and become the important content of research.
Summary of the invention
It is an object of the invention to overcome the deficiencies in the prior art, a kind of StaMPS Algorithm parallelization processing method based on Hadoop is provided, achieve the combination of Hadoop Yu StaMPS algorithm, it is achieved thereby that the storage of mass data and calculating, enhance autgmentability and the availability of system.
In order to achieve the above object, the technical solution used in the present invention is:
A kind of StaMPS Algorithm parallelization processing method based on Hadoop, described Hadoop includes HDFS and MapReduce, including:
Essence registration, runs in Hadoop cloud platform, it is achieved the registration of image, and registration unit serial process is transform as multi-host parallel；
PS(full name Permanent Scatterers, permanent scattering object) analyze, run in Hadoop cloud platform, it is achieved the selection of PS point, simplify, the process of unit serial order is transform as multi-host parallel.
It is preferred that described essence registration includes:
Auxiliary image is to the essence registration of main image: for being set with oneself less than n(by all baselines with main image, such as 100) the auxiliary image of rice is directly registrable main image space, and the numerical value of n according to circumstances sets；
Auxiliary image is to the essence registration of auxiliary image: for being set with oneself more than n(by the baseline with main image, such as 100) Image registration of rice is to the space of the most nearest m auxiliary image, certainly the present invention is not limited to the space of 2～4 auxiliary images, adjusting according to specific needs, the numerical value of n, m according to circumstances sets.Such as, the space of 2～4 auxiliary images, certain present invention is not limited to the space of 2～4 auxiliary images, adjusts according to specific needs.
Preferably, described auxiliary image, to the smart registration of main image, transformation for circulation, is used for that loop iteration catalogue process before is transform as MapReduce parallel, each Map(accepts a key-value pair, produces one group of middle key-value pair) process a catalogue data being independent of each other.
It is preferred that described auxiliary image is to the essence registration of auxiliary image, transforming dual for circulation, parallel for loop iteration catalogue process before is transform as MapReduce, each Map processes a catalogue data being independent of each other.
Being made up of it is preferred that described PS analyzes module, described module includes:
Data load-on module, the form needed for data are converted into PS analyze, and store data into matlab space；
Calculate time coherence coefficient module, the time coherence coefficient of each candidate point in iterative computation interference pattern；
PS point selection module, the non-PS point pixel maximum of proportion in total pixel according to setting is tried to achieve time coherence coefficient threshold adaptively, thus is selected PS point；
PS point simplifies module, rejects owing to being made the interferometric phase noise point more than predetermined threshold value by neighbourhood effect；
Space incoherent Correction of Errors module, for carrying out the incoherent Correction of Errors in space, including the incoherent error of space noncoherent angle of sight error and the space relevant with main image to the phase place being wound around.
It is preferred that StaMPS Algorithm parallelization processing method based on Hadoop comprises the steps:
Essence registration:
Step 101. carries out piecemeal, a corresponding auxiliary shadow directory of Map auxiliary shadow directory, carries out the auxiliary image registration to main image；
Step 102. accepts a key at Reduce(, and a relevant class value, this class value is merged and produces the value (usual only one of which or zero value) that one group of scale is less) hold the auxiliary image generating needs registration to record, described auxiliary image is constituted by the auxiliary image registrated and the auxiliary image being registered；
The input as next MapReduce chosen as requested by the n that step 103. generates Reduce auxiliary image；
The each Map of step 104. processes the auxiliary image registration to auxiliary image；
PS analyzes:
Step 201. carries out deblocking, extracts PS candidate point, and each Map processes a blocks of data；
Step 202. data load, and data are converted into PS and analyze the form needed, and store data into matlab space；
Step 203. calculates time coherence coefficient, the time coherence coefficient of each candidate point in iterative computation interference pattern (using the pattern being made up of interference band and black arm interfering ball to be presented when observing heterogeneous body jewel under cross-polarized light, it is due to the summation through delustring produced by the tapered polarization light of crystal Yu interference effect)；
Step 204.PS point selection, the non-PS point pixel maximum of proportion in total pixel according to setting is tried to achieve time coherence coefficient threshold adaptively, thus is selected PS point；
Step 205.PS point is simplified, and rejects owing to being made the interferometric phase noise point more than predetermined threshold value by neighbourhood effect；
The incoherent Correction of Errors in step 206. space, carries out the incoherent Correction of Errors in space, including the incoherent error of space noncoherent angle of sight error and the space relevant with main image to the phase place being wound around；
Step 207. is called matlab and is merged；
Step 208. carries out phase unwrapping, spatial coherence Correction of Errors, denoising phase operation.
Compared with prior art, the invention has the beneficial effects as follows: essence registration is achieved parallel by serial transformation, the PS analysis phase achieves parallel from step 1 to step 7, combination by Hadoop and StaMPS algorithm, StaMPS algorithm process efficiency is increased exponentially, adapt to storage and the calculating of the mass data of big data age, improve availability and the autgmentability of system.
Accompanying drawing explanation
Fig. 1 is the system architecture diagram of the present invention；
Fig. 2 is the essence registration parallel processing flow chart of the present invention；
Fig. 3 is that the PS of the present invention analyzes parallel processing flow chart.
Detailed description of the invention
Idea of the invention is that and overcome the deficiencies in the prior art, a kind of StaMPS Algorithm parallelization processing method based on Hadoop is provided, StaMPS is a kind of novel PSInSAR method, unlimited storage can be realized and calculate, because Hadoop has high fault-tolerant, high reliability, enhanced scalability, high acquired, the features such as high-throughput, by analyzing StaMPS algorithm process flow process and repetition test, have found bottleneck (the most time-consuming two parts of StaMPS algorithm, account for about the 96% of whole handling process) place, simultaneously it has also been found that these two parts meet the condition of parallel processing, therefore these two parts are got up parallel, it is greatly improved the efficiency of StaMPS algorithm process.Hadoop is a Distributed Computing Platform of increasing income.With Hadoop distributed file system (HDFS, Hadoop Distributed Filesystem) and the realization of increasing income of MapReduce(Google MapReduce) it is that the Hadoop of core has provided the user the distributed basis framework that system bottom details is transparent.
It is described in detail referring to the drawings below in conjunction with embodiment, in order to technical characteristic and advantage to the present invention are interpretated more in-depth.
The system block diagram of the present invention is as it is shown in figure 1, include client (image file), Hadoop cloud platform (also for Hadoop cluster, including HDFS and MapReduce) and basic resource (StaMPS algorithm process software)
StaMPS Algorithm parallelization processing method based on Hadoop disclosed in this invention, described Hadoop includes HDFS and MapReduce, including:
Essence registration, runs in Hadoop cloud platform, it is achieved the registration of image, and registration unit serial process is transform as multi-host parallel；
PS analyzes, and runs in Hadoop cloud platform, it is achieved the selection of PS point, simplify, and the process of unit serial order is transform as multi-host parallel.Because the reading of each auxiliary image Precise Orbit information, all isolated operations in each auxiliary image presss from both sides are generated with the registration of other images, resampling, differential interferometry figure, it is independent of each other with other images, meet the condition of parallel processing, therefore it can also be carried out parallel processing, at stamps(5,5) call at ending merge merges process.
It is preferred that described essence registration includes:
Auxiliary image is to the essence registration of main image: be directly registrable main image space for all baselines with main image are less than the auxiliary image of 100 meters；Script is processed by analyzing it, single-threaded weight for searching loop image file catalogue (the image file catalogue a being independent of each other) sequential processes is transform as and is processed a catalogue parallel processing by each Map of MapReduce, merged by Reduce again, substantially reduce its process time, improve treatment effeciency.
Auxiliary image is to the essence registration of auxiliary image: for can setting with oneself more than n(with the baseline of main image, such as 100) Image registration of rice is to the space of 2～4 the most nearest auxiliary images.The each Map being transform as MapReduce by the file directory order traversal read-write recirculated two processes a catalogue parallel processing, then is merged by Reduce, substantially reduces its process time, improves treatment effeciency.
It is preferred that described auxiliary image is to the essence registration of main image, transformation for circulation, i.e. loop iteration catalogue process before being transform as MapReduce parallel, each Map processes a catalogue data being independent of each other.
It is preferred that described auxiliary image is to the essence registration of auxiliary image, transforming dual for circulation, i.e. loop iteration catalogue before is processed and transform MapReduce as parallel, each Map processes a catalogue data being independent of each other.
Being made up of it is preferred that described PS analyzes module, described module includes:
Data load-on module, the form needed for data are converted into PS analyze, and store data into matlab space；
Calculate time coherence coefficient module, the time coherence coefficient of each candidate point in iterative computation interference pattern；
PS point selection module, the non-PS point pixel maximum of proportion in total pixel according to setting is tried to achieve time coherence coefficient threshold adaptively, thus is selected PS point；
PS point simplifies module, rejects owing to being made the interferometric phase noise point more than predetermined threshold value by neighbourhood effect；
Space incoherent Correction of Errors module, for carrying out the incoherent Correction of Errors in space, including the incoherent error of space noncoherent angle of sight error and the space relevant with main image to the phase place being wound around.
It is preferred that StaMPS Algorithm parallelization processing method based on Hadoop comprises the steps:
As in figure 2 it is shown, essence registration:
Step 101. carries out piecemeal, a corresponding auxiliary shadow directory of Map auxiliary shadow directory, carries out the auxiliary image registration to main image；
Step 102. generates at Reduce end needs the auxiliary image of registration to record；
The input as next MapReduce chosen as requested by the n that step 103. generates Reduce auxiliary image；
The each Map of step 104. processes the auxiliary image registration to auxiliary image；
As it is shown on figure 3, PS analyzes:
Step 201. carries out deblocking, extracts PS candidate point, and each Map processes a blocks of data；
Step 202. data load, and data are converted into PS and analyze the form needed, and store data into matlab space；
Step 203. calculates time coherence coefficient, the time coherence coefficient of each candidate point in iterative computation interference pattern；
Step 204.PS point selection, the non-PS point pixel maximum of proportion in total pixel according to setting is tried to achieve time coherence coefficient threshold adaptively, thus is selected PS point；
Step 205.PS point is simplified, and rejects owing to being made the interferometric phase noise point more than predetermined threshold value by neighbourhood effect；
The incoherent Correction of Errors in step 206. space, carries out the incoherent Correction of Errors in space, including the incoherent error of space noncoherent angle of sight error and the space relevant with main image to the phase place being wound around；
Step 207. is called matlab and is merged；
Step 208. carries out phase unwrapping, spatial coherence Correction of Errors, denoising phase operation.
Above content is to combine concrete preferred embodiment further description made for the present invention, should not assert the present invention be embodied as be confined to described above.For those skilled in the art, without departing from the inventive concept of the premise, it is also possible to make some simple deduction or replace, within the protection domain that the claim being regarded as being submitted to by the present invention determines.
Claims (5)
1. a StaMPS Algorithm parallelization processing method based on Hadoop, described Hadoop bag
Include HDFS and MapReduce, it is characterised in that including:
Essence registration, runs in Hadoop cloud platform, it is achieved the registration of image, registration unit string
Row process transform multi-host parallel as；
PS analyzes, and runs in Hadoop cloud platform, it is achieved the selection of PS point, simplify, unit
Serial order processes and transform multi-host parallel as；
Comprise the steps:
Essence registration:
Step 101. carries out piecemeal, a corresponding auxiliary shadow directory of Map auxiliary shadow directory, enters
The auxiliary image of row is to the registration of main image；
Step 102. Reduce end generate need registration auxiliary image to record, described auxiliary image pair
It is made up of the auxiliary image registrated and the auxiliary image being registered；
The N number of auxiliary image that step 103. generates Reduce is chosen as requested as the next one
The input of MapReduce, the numerical value of N according to circumstances sets；
The each Map of step 104. processes the auxiliary image registration to auxiliary image；
PS analyzes:
Step 201. carries out deblocking, extracts PS candidate point, and each Map processes a blocks of data；
Step 202. data load, and data are converted into PS and analyze the form needed, and data stored
To matlab space；
Step 203. calculates time coherence coefficient, the time phase of each candidate point in iterative computation interference pattern
Responsibility number；
Step 204.PS point selection, according to the non-PS point pixel proportion in total pixel set
Maximum tries to achieve time coherence coefficient threshold adaptively, thus selects PS point；
Step 205.PS point is simplified, and rejects owing to being made interferometric phase noise be more than by neighbourhood effect
The point of predetermined threshold value；
The incoherent Correction of Errors in step 206. space, carries out the incoherent error in space to the phase place being wound around and changes
Just, including the incoherent error of space noncoherent angle of sight error and the space relevant with main image；
Step 207. is called matlab and is merged；
Step 208. carries out phase unwrapping, spatial coherence Correction of Errors, denoising phase operation.
StaMPS Algorithm parallelization process side based on Hadoop the most according to claim 1
Method, it is characterised in that described essence registration includes:
Auxiliary image is to the essence registration of main image, for the baseline of all and main image is less than the auxiliary of n rice
Image is directly registrable main image space, and the numerical value of n according to circumstances sets；
Auxiliary image is to the essence registration of auxiliary image, for being joined more than the image of n rice by the baseline with main image
The accurate space to m the most nearest auxiliary image, the numerical value of n, m according to circumstances sets.
StaMPS Algorithm parallelization process side based on Hadoop the most according to claim 2
Method, it is characterised in that: the essence registration of described auxiliary image to main image, transformation for circulation, i.e. it
Front loop iteration catalogue processes and transform MapReduce as parallel, and each Map processes a most not shadow
The catalogue data rung.
StaMPS Algorithm parallelization process side based on Hadoop the most according to claim 2
Method, it is characterised in that: the essence registration of described auxiliary image to auxiliary image, transform dual for circulation, i.e.
Loop iteration catalogue before being processed and transform MapReduce as parallel, each Map processes one mutually
The catalogue data not affected.
StaMPS Algorithm parallelization process side based on Hadoop the most according to claim 1
Method, it is characterised in that described PS analyzes and is made up of module, and described module includes:
Data load-on module, the form needed for data are converted into PS analyze, and data are deposited
Storage is to matlab space；
Calculate time coherence coefficient module, the time phase of each candidate point in iterative computation interference pattern
Responsibility number；
PS point selection module, according to the non-PS point pixel maximum of proportion in total pixel set
Value tries to achieve time coherence coefficient threshold adaptively, thus selects PS point；
PS point simplifies module, for rejecting owing to being made interferometric phase noise more than pre-by neighbourhood effect
If the point of threshold value；
Space incoherent Correction of Errors module, changes for the phase place being wound around is carried out the incoherent error in space
Just, including the incoherent error of space noncoherent angle of sight error and the space relevant with main image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410152271.9A CN103903272B (en) | 2014-04-16 | 2014-04-16 | A kind of StaMPS Algorithm parallelization processing method based on Hadoop |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410152271.9A CN103903272B (en) | 2014-04-16 | 2014-04-16 | A kind of StaMPS Algorithm parallelization processing method based on Hadoop |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103903272A CN103903272A (en) | 2014-07-02 |
| CN103903272B true CN103903272B (en) | 2016-09-07 |
Family
ID=50994579
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410152271.9A Expired - Fee Related CN103903272B (en) | 2014-04-16 | 2014-04-16 | A kind of StaMPS Algorithm parallelization processing method based on Hadoop |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103903272B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109801319B (en) * | 2019-01-03 | 2020-12-01 | 杭州电子科技大学 | Hierarchical graph grouping and registering method based on Hadoop parallel acceleration |
| CN115511925B (en) * | 2022-09-26 | 2023-11-03 | 成都理工大学 | StaMPS parallel optimization processing method and product based on pipeline communication mechanism |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101350098A (en) * | 2008-08-19 | 2009-01-21 | 黄建军 | Method, system and equipment for processing image |
| CN101604018A (en) * | 2009-07-24 | 2009-12-16 | 中国测绘科学研究院 | High-definition remote sensing image data disposal route and system thereof |
| CN102662610A (en) * | 2012-04-20 | 2012-09-12 | 中国科学院遥感应用研究所 | Remote-sensing data management system and method |
| CN103281351A (en) * | 2013-04-19 | 2013-09-04 | 武汉方寸科技有限公司 | Cloud server platform capable of efficiently processing and analyzing remote sensing data |
-
2014
- 2014-04-16 CN CN201410152271.9A patent/CN103903272B/en not_active Expired - Fee Related
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101350098A (en) * | 2008-08-19 | 2009-01-21 | 黄建军 | Method, system and equipment for processing image |
| CN101604018A (en) * | 2009-07-24 | 2009-12-16 | 中国测绘科学研究院 | High-definition remote sensing image data disposal route and system thereof |
| CN102662610A (en) * | 2012-04-20 | 2012-09-12 | 中国科学院遥感应用研究所 | Remote-sensing data management system and method |
| CN103281351A (en) * | 2013-04-19 | 2013-09-04 | 武汉方寸科技有限公司 | Cloud server platform capable of efficiently processing and analyzing remote sensing data |
Non-Patent Citations (3)
| Title |
|---|
| A new method for measuring deformation on volcanoes and other natural terrains using InSAR persistent scatterers;Andrew Hooper et al.;《Geophysical Research Letters》;20041210;第31卷;第1-5页 * |
| Research and Implementation on Spatial Data Storage and Operation Based on Hadoop Platform;Yonggang Wang et al.;《2010 Second IITA International Conference on Geoscience and Remote Sensing》;20100831;第275页第2栏第5段第5-7行，第276页第2栏第4段第1-5行，第277页第2栏第1-2段 * |
| StaMPS/MTI Manual Version 3.3b1;Andy Hooper et al.;《UNIVERSITY OF LEEDS》;20130912;正文第1页第3段第1-3行，第13页第4段第1-2行，第17-19页第2段 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103903272A (en) | 2014-07-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10885467B2 (en) | Differentially private iteratively reweighted least squares | |
| CN109100786B (en) | Method and device for determining quality factor of depth domain | |
| US10915531B2 (en) | Algebraic query language (AQL) database management system | |
| CN105030201A (en) | Correction method and system of sweep frequency OCT digital phases based on information on static regions | |
| US11734235B2 (en) | Automatic drift detection and handling | |
| US10217225B2 (en) | Distributed processing for producing three-dimensional reconstructions | |
| CN114578356A (en) | Distributed scatterer deformation monitoring method, system and equipment based on deep learning | |
| CN103558599A (en) | Complex heterogeneity forest stand mean height estimating method based on multisource remote sensing data | |
| CN103903272B (en) | A kind of StaMPS Algorithm parallelization processing method based on Hadoop | |
| CN107392239A (en) | A kind of K Means algorithm optimization methods based on Spark computation models | |
| CN117897720A (en) | Denoising depth data for low signal pixels | |
| Wang et al. | Infrared and visible image fusion via decoupling network | |
| Li et al. | AEFusion: A multi-scale fusion network combining Axial attention and Entropy feature Aggregation for infrared and visible images | |
| Wu et al. | Fast, accurate normalized cross-correlation image matching | |
| Feng et al. | Multi-scale change monitoring of water environment using cloud computing in optimal resolution remote sensing images | |
| CN102289715A (en) | Method for adaptively denoising and modeling wavelet neural network based on forward linear prediction (FLP) | |
| CN108921896A (en) | A kind of lower view vision compass merging dotted line feature | |
| CN104215339B (en) | Wavefront reconstruction system and wavefront reconstruction method based on continuous far fields | |
| Liu et al. | Study on the prediction of cotton yield within field scale with time series hyperspectral imagery | |
| WANG et al. | Simultaneous-source wavefield separation based on adaptive Wiener threshold in Contourlet domain | |
| CN109190506A (en) | It is a kind of based on core is sparse and the EO-1 hyperion object detection method of space constraint | |
| Wang et al. | Stream-computing of high accuracy on-board real-time cloud detection for high resolution optical satellite imagery | |
| Barajas et al. | Benchmarking parallel k-means cloud type clustering from satellite data | |
| Li et al. | Contracting medial surfaces isotropically for fast extraction of centred curve skeletons | |
| CN107894942A (en) | The monitoring method and device of tables of data visit capacity |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20160907 |