CN106453618A - Remote sensing image processing service cloud platform system based on G-Cloud cloud computing - Google Patents
Remote sensing image processing service cloud platform system based on G-Cloud cloud computing Download PDFInfo
- Publication number
- CN106453618A CN106453618A CN201611005463.2A CN201611005463A CN106453618A CN 106453618 A CN106453618 A CN 106453618A CN 201611005463 A CN201611005463 A CN 201611005463A CN 106453618 A CN106453618 A CN 106453618A
- Authority
- CN
- China
- Prior art keywords
- cloud
- remote sensing
- data
- service
- platform
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a remote sensing image processing service cloud platform system based on G-Cloud cloud computing, which comprises a G-Cloud cloud computing platform, a remote sensing image processing cloud platform and a remote sensing processing cloud service mode and demonstration application, wherein the G-Cloud cloud computing platform is used for implementing a physical resource management, safety management and basic management platform and a service platform and providing support for the remote sensing processing cloud platform; the remote sensing image processing cloud platform is established on the basis of the G-Cloud cloud computing platform, and comprises a remote sensing processing cloud system structure, a raster data management system facing remote sensing processing cloud, and an image processing and high-quality product production algorithm facing remote sensing processing cloud; and the remote sensing processing cloud service mode and demonstration application is used for solving the problems of actual use of a user and demonstration verification of the system. According to the remote sensing image processing service cloud platform system disclosed by the invention, by design of a remote sensing cloud system architecture, computing of data from single-node computing from multi-node parallel computing is implemented, and computing efficiency of the data is improved; and by adopting a cloud service data sharing mode, real-time network sharing of the remote sensing data is implemented, and the problem of fussy user data sharing is solved.
Description
Technical field
The invention belongs to Computer Applied Technology field is and in particular to a kind of remote sensing images based on G-Cloud cloud computing
Process service cloud platform system.
Background technology
With China's remote sensing and land observation system gradual perfection, remote sensing application starts gradually from professional application department deeply
To individual application and public service field, and the mode of operation of current single-set operation and professional system configuration, generally adopted
In complicated model and handling process so that user be difficult to make full use of data resource and obtain convenient, intuitively service, and base
Exactly solve remote sensing information resource-sharing and the strongest instrument of service in the process of cloud and service mode, can effectively reduce to making
User and the threshold requirement of use environment.
International GIS software leader ESRI company is proposed this cloud platform of ArcGIS Online in November, 2009, should
Platform has run online shared environment, online business analysis, resource center and various online drawing service etc..ESRI is all of
Product and based on ArcGIS exploitation application can access on ArcGIS Online provide content and service.ArcGIS
Online can regard a kind of WEB GIS platform and infrastructure as, and it is by infrastructure layer, podium level and three layers of application layer
Secondary composition (ESRI, 2009).Many on limits resources are also simply simply mounted to by ArcGIS Online at present
The upper user for needing of Amazon S3 (Simple Storage Service) and GIS fan carry out online browse and download,
The data being provided that and function phase, to being all fairly simple, can not be taken and come in the middle of the application of reality, major part also stops
In Demo and demonstration stage.In July, 2010, ESRI has issued the GIS new product ArcGIS 10 supporting cloud framework, and it is permissible
Directly it is deployed on cloud computing platform, high in the clouds is served to the management of spatial data, analysis and processing function.ArcGIS's 10
It is achieved in that the infrastructure powerful using cloud computing platform, by original software hosting in cloud platform, externally provide number
According to and software service.ArcGIS 10 is the cloud computing application walked in GIS-Geographic Information System industry foremost at present.
At present both at home and abroad research and utilization cloud computing technology processing spatial information, most typical technology path is using publicly-owned
The hardware environment that cloud computing platform provides, by original software hosting in platform, externally provides data and software service,
ArcGIS Online and ArcGIS 10 of ESRI just employs such implementation.This mode does not consider cloud computing basis
The operation logic of facility and construction characteristic it is not necessary to big change is done to original software, its original data storage, index and place
Reason technology does not all change, and still continues to use the technology mode of original GIS software, and it is only that this also leads to it cannot make full use of cloud computing
Special distributed computing architecture and the advantage of storage system, the not innovation of essence and lifting on software design scheme, with
When also cannot solve operation problem in private clound for the software.
Content of the invention
It is an object of the invention to provide a kind of remote sensing image processing service cloud platform system based on G-Cloud cloud computing,
Solve the problems, such as that prior art computational efficiency is low, data sharing is difficult.
The technical solution adopted in the present invention is, the remote sensing image processing service cloud platform system based on G-Cloud cloud computing
System, including：
G-Cloud cloud computing platform, is used for realizing physical resource management, safety management, basic management platform and services flat
Platform, processing cloud platform for remote sensing provides support, and is extended and optimizes for effectively realizing remote sensing image processing cloud platform；
Remote sensing image processing cloud platform, sets up on the basis of G-Cloud cloud computing platform, processes cloud system knot including remote sensing
Structure, the raster data management system processing cloud towards remote sensing, the image procossing processing cloud towards remote sensing and high-grade products production
Algorithm；
Remote sensing processes cloud service pattern and Demonstration Application, and for solving, user is actually used and system demonstration is verified.
The feature of the present invention also resides in：
Remote sensing processes cloud architecture and is logically divided into host environment and resource, virtual environment, the system core, exploitation and
Use environment, cloud computing remote sensing cloud service center；
Host environment and resource：It is made up of the various resources that wide area is distributed, be the basic thing constituting all nodes of remote sensing cloud
Reason and the general name of logic entity, including the computer of node, storage system, main equipment instrument, operating system kernel, in cluster
Between part；
Virtual environment：Including the platform virtualization calculating with operating system, particular system resource internal memory, storage, network system
System resource virtualizing, and application virtualization；
The system core：For connecting the systems soft ware of cloud resource；
Cloud computing remote sensing cloud service center：The various remote sensing applications of exploitation on remote sensing cloud system layer are provided, support empty
Planization remote desktop, zero-configuration client, provide software to be service function, multi-user leases multiple application software use patterns.
Cloud computing remote sensing cloud service center：The various remote sensing applications of exploitation on remote sensing cloud system layer are provided, support empty
Planization remote desktop, zero-configuration client, provide software to service (SaaS) function, multiple application softwaries such as multi-user's lease use mould
Formula.
The raster data management system processing cloud towards remote sensing includes magnanimity remote sensing raster data cloud storage structural model, distant
The organizational structure of sense database data and remote sensing metadata standard structure；
Magnanimity remote sensing raster data cloud storage structural model：Comprise accumulation layer, basic management layer, application-interface layer and access
Layer, this structural model is applied to the storage of data and the management that data scale reaches more than 100TB rank；
The organizational structure of remotely-sensed data database data：It is right to be come using these three levels of original data layer, organized layer and application layer
Complicated remotely-sensed data carries out organizing, manages；
Remote sensing metadata standard structure：For for different data-storage system provide unified data access interface and
The standard data structure of API specification.
The algorithm of the image procossing and high-grade products production that process cloud towards remote sensing comprises general image parallel processing algorithm
Pattern, the remote sensing image processing algorithm pattern based on cloud computing mode；
In general image parallel processing algorithm pattern, the transmission of task data passes through xml document with scheduling, forms standard
Change form, possess cross-platform ability, be applicable in the hind computation environment of cloud process；
In the remote sensing image processing algorithm pattern based on cloud computing mode, adopt and be combined with Map-Reduce based on MPI
Parallel Implementation mode, using memory sharing data pre-read, magnetic disc i/o operation is optimized, can once read, enter more
Journey multithreading is shared, and using component pattern, keeps " loose coupling " of Business Logic or business support layer function assembly, algorithm and
Module individual packages.
Remote sensing processes cloud service pattern and adopts SOA framework, according to OGC with remote sensing process cloud service pattern in Demonstration Application
Web service system is built, and the remote sensing image processing function in system is realized in the way of Web service and is packaged with WPS, right
All provide in accordance with the specification of OGC in the required data of service, both service chaining and execution are realized with working flow mode；
Whole system is divided by function as 3 levels：Data resource service layer, function treatment service layer and application logic
Layer；
Data resource service layer is made up of OGC WCS, OGC WMS, OGC WFS and OGC CSW, and wherein WCS provides grid
Image services, and can return the remote sensing image data as GeoTiff form；CSW provides the metadata catalog service of mass data；
WFS and WMS returns vector map datum respectively；
Function treatment service layer contains the various functional modules to geographical spatial data handling, and according to WPS specification pair
These modules are packed；
Application logic control layer comprises 3 modules：Workflow storehouse, flow definition editor module and flow processing module.
Remote sensing processes cloud service pattern and includes just penetrating production Demonstration Application, laser thunder with Demonstration Application in Demonstration Application
Reach cloud data DEM production Demonstration Application；
Just penetrating production Demonstration Application is to process global TM base reference Image Database, the Yong Hujian of cloud management using remote sensing
Vertical control point storehouse and the 90 Miho Dockyard EM assistance datas in the whole world, call in process cloud algorithms library, are just penetrating correction and be related to algorithm, realize
Just penetrating the production of product；
In laser radar point cloud data DEM production Demonstration Application laser radar point cloud data be related to a little very intensive,
Data volume is big, will filter acquisition high quality DEM, needs complicated computing.This Demonstration Application utilizes remote sensing to process the complete of cloud management
90 Miho Dockyard EM assistance datas of ball, call process cloud algorithms library point cloud filtering algorithm, realize the production of DEM product.
The invention has the beneficial effects as follows：The remote sensing image processing service cloud platform system based on G-Cloud cloud computing for the present invention
System, by the design of remote sensing cloud architectural framework, realizes data and calculates multi-node parallel calculating by single node, improve the fortune of data
Calculate efficiency；By using cloud service Data share model, the real-time network realizing remotely-sensed data is shared, and solves user data and shares
Loaded down with trivial details problem.
Brief description
Fig. 1 is the system level graph of a relation that in cloud platform system of the present invention, remote sensing processes cloud service pattern.
Specific embodiment
The present invention is described in detail with reference to the accompanying drawings and detailed description.
The remote sensing image processing service cloud platform system based on G-Cloud cloud computing for the present invention, including：
G-Cloud cloud computing platform, is used for realizing physical resource management, safety management, basic management platform and services flat
Platform, processing cloud platform for remote sensing provides support, and is extended and optimizes for effectively realizing remote sensing image processing cloud platform；System
Middle hardware environment is based on G-Cloud platform bottom, and each server is divided into physical server (management node), physics according to function
Server (calculate node), data server etc..
Remote sensing image processing cloud platform, sets up on the basis of G-Cloud cloud computing platform, processes cloud system knot including remote sensing
Structure, the raster data management system processing cloud towards remote sensing, the image procossing processing cloud towards remote sensing and high-grade products production
Algorithm；
First, remote sensing processes cloud architecture
Remote sensing processes cloud architecture and is logically divided into host environment and resource, virtual environment, the system core, exploitation and
Use environment, the several level of cloud computing remote sensing cloud service center, hierarchical division method can by limit every layer complexity and
Effectively reduce the overall complexity of system, and reduce the overseas publicity that internal change impacts；Service-oriented method reduces
Stiffness of coupling between functional part, makes system dynamics replace and be expanded into possibility.By core system calling system storehouse, transport
Row supports and development kit provides for development of user and realizes unrelated interface with system, supports the exploitation of other remote sensing cloud parts
And operation.
Host environment and resource：It is made up of the various resources that wide area is distributed, be the basic thing constituting all nodes of remote sensing cloud
Reason and the general name of logic entity, including the computer of node, storage system, main equipment instrument, operating system kernel, in cluster
Between part；Cloud provider can provide resource lease service.
Virtual environment：Including the platform virtualization calculating with operating system, particular system resource internal memory, storage, network system
System resource virtualizing, and application virtualization；Hypervisor, virtual machine control block, efficient I/O virtualization etc. are provided.Support
The connectedness of elasticity, virtual resources topological structure and resource.
The system core：For connecting the systems soft ware of cloud resource.Based on the system core, facilities services layer provides cloud resource
Description, positioning, configuration, scheduling and corresponding security mechanism；There is provided user resources, authentication, Single Sign-On, access is awarded
The functions such as power, book keeping operation；Message Processing, local policy, data duplication, data transfer, event handling, system monitoring etc. is provided to support
Function；Offer services and interacts between accessing, service and servicing and communicate, and protocol conversion, route, access guarantee, message format turn
The function such as change；Guarantee service position is transparent, protocol transparent, form are transparent, the binding that decoupling service is realized with technology.
Exploitation and use environment：For providing platform to service the function of (PaaS), provide remote sensing cloud runtime, remote sensing
Corresponding instrument of cloud etc., supports diversification and customizable software platform service, supports that resource accesses, the collaborative of remote sensing cloud application opens
Send out etc..
Cloud computing remote sensing cloud service center：The various remote sensing applications of exploitation on remote sensing cloud system layer are provided, support empty
Planization remote desktop, zero-configuration client, provide software to service (SaaS) function, multiple application softwaries such as multi-user's lease use mould
Formula.
2nd, the raster data management system of cloud is processed towards remote sensing
Towards remote sensing process cloud raster data management system be directed to extendible data, services framework, data storage with
The aspects such as data transfer are optimized, and devise magnanimity remote sensing raster data cloud storage structural model, remotely-sensed data database data
Organizational structure and remote sensing metadata standard structure.
(1) magnanimity remote sensing raster data cloud storage structural model
When storage and management that the core of cloud computing system computing and process is mass data, just need in cloud computing system
Configure substantial amounts of storage device, at this moment cloud computing system is transformed into as a cloud storage system.Project pass through cluster application,
The research of the correlation technique such as grid or distributed file system, by network in a large number various types of storage device lead to
Cross application software and gather collaborative work, jointly externally provide remotely-sensed data storage distant with a magnanimity of Operational Visit function
Sense raster data cloud storage service platform, the structural model of this platform comprises accumulation layer, basic management layer, application-interface layer and visit
Ask layer.
Accumulation layer
Accumulation layer is the most basic part of cloud storage.Storage device in cloud storage often substantial amounts and be distributed how different
Region, is linked together by wide area network, the Internet or FC fiber channel network each other.It is one on storage device
Uniform storage device management system, it is possible to achieve the logical Virtual management of storage device, multi-link redundancy management, and hardware
The condition monitoring of equipment and Breakdown Maintenance.
Basic management layer
Basic management layer is the most crucial part of cloud storage, is also the part being difficult to most in cloud storage.Basic management
Layer, by technology such as cluster, distributed file system and grid computings, is realized collaborative between multiple storage devices in cloud storage
Work, makes multiple storage devices can externally provide same service, and provides more higher greatly more preferable data access performance.
CDN content dissemination system, data encryption technology ensure that the data in cloud storage will not be accessed by undelegated user, meanwhile,
Can ensure that by various data backups and disaster tolerance technology and measure the data in cloud storage will not be lost it is ensured that cloud storage itself
Safety and stable.
Application-interface layer
Application-interface layer is cloud storage part the most flexible and changeable.Different cloud storage units of operation can be according to actual industry
Service type, develops different application service interfaces, provides different application services.Such as raster data application platform, remotely count
According to back-up application platform etc..
Access layer
Any one authorized user can log in cloud storage system by the public application interface of standard, enjoy cloud and deposit
Storage service.Cloud storage unit of operation is different, and the access type that cloud storage provides and addressing meanses are also different.
(2) organizational structure of remotely-sensed data database data
Remotely-sensed data is probably digitized it is also possible to non-digitalization；It is probably to have passed through data management system pipe
Reason it is also possible to still be in what discrete state did not managed in an efficient way.The application adopts original data layer, organized layer and answers
Organize remotely-sensed data with three level of layer；
Original data layer
The derived data layer of Remote Sensing Database.In original data layer, data is probably digitized it is also possible to nonnumeric
Change；Be probably by data management system manage it is also possible to still be in what discrete state did not managed in an efficient way.
For Remote Sensing Database, this layer is basis.Content Organizing work on this layer is main to be included：Data collection；According to
The data model data specification of organized layer's definition is processed to data arranging；And set up relevant data and organized layer's word bank
In about the mapping of data and transformational relation.The data of original data layer through Automatic Program conversion and imports or manual entry,
Enter the word bank of organized layer.
Data cleansing
There may exist substantial amounts of dirty data in initial data, need to dig using relevant technology such as mathematical statisticss, data
Dirty data is changed into the data meeting quality of data requirement by pick or predefined data cleansing rule.Undesirable data
Mainly there are the data three major types of incomplete data, the data of mistake and repetition.
Data conversion
Initial data tends not to directly correspond in the middle of the model of organized layer, and needs a series of conversion and computing.
Three common class data conversion situations include：Field type is changed：Refer to field class in transformation process for the implication identical field
Type there occurs change；Merge：Multiple fields of initial data, after arithmetical operation or logical operationss, form in word bank
Individual field, merges the connection being likely to including character type field value；Split：Refer to that in word bank, a field is transported through arithmetic or logic
After calculation, correspond to the multiple fields in the middle of word bank, or this field splits into after several substrings for character type field value, every height
String one of works as field corresponding to word bank.
Data maps
After the model of organized layer determines, initial data should set up the mapping ruler and model between.The building of mapping ruler
Stand and include two parts content：The foundation of mapping relations and the foundation of transformational rule.
Set up mapping relations：Mapping ruler is not simple corresponding relation, according to the source and destination structure of actual migration, also
Fractionation, merging of field etc. may be comprised.For the data model expressed with entity relationship model, mapping is embodied in entity
Aspect and attribute level.Entity maps：By attribute source and the situation about splitting of entites in data model, source table and purpose table exist
One-to-one mapping can be divided on quantity, one-to-many maps, many-one maps, multi-to-multi maps.Field maps：Relation can be divided into
3 kinds, i.e. directly mapping, major key mapping and external key mapping.Wherein：Major key mapping is to ensure that in raw data base, main external key is about
Bundle is retained in remotely-sensed data character library, and external key mapping is to ensure that in Remote Sensing Database character library, foreign key field can be from initial data
In the table of storehouse, corresponding field correctly migrates.Direct mapping is exactly that the field in speciality dictionary table maps directly to the word in data model table
Duan Shang, the computing such as does not split, merges.Type mapping：There are different tools in data type in the middle of different implementation methods
Body surface reaches, and the two ends of Type mapping are probably a data type under this implementation method it is also possible to a data type adds
Format constraints.
The expression of Mapping and Converting：Each mapping relations can be expressed as a source, object and the tlv triple producing formula.
Object：The set of mapped end object, object can be base level (field aspect, variable) can also be base level
The set of object composition, if there is the object of non-basic level, its mapping ruler finally should implement to base level；Source：
Each object is in the corresponding Data Source of source；Production formula：The Data Source of source all can be processed according to production formula, and shape
Become to meet the contents of object of destination end data model requirement.
Organized layer
Preserve, manage and provide this layer of data of service in Remote Sensing Database.In organized layer, data passes through remotely-sensed data
Word bank preserves and manages, and meets relevant data standard, and by Remote Sensing Database data service system for the user discover that and visiting
Ask, and research application service system is supplied to by Web Service or other modes and utilize.This layer is Remote Sensing Database
Core.For Remote Sensing Database is built, the Content Organizing work on this layer is main to be included：Determine in Remote Sensing Database
Hold framework and word bank data model；Determine relevant data standard；For word bank logging data；Set up data classified catalogue；Create
Metadata of word bank, research application service system etc. etc..
Organized layer defines the model of some special topic word banks, higher level is carried out to analysis object data one complete,
Consistent description, can complete, uniformly portray each item data of each research field involved by analysis object, and between data
Contact, organize data into a complete specific analysis data environment.
The data of organized layer mostlys come from the already present initial data of industry, and the model of design organization layer seeks to design such as
What obtains completely consistent data from existing data source, how obtained data is carried out changing, to be recombinated, comprehensive, such as
What effectively improves efficiency and accuracy of data analysiss etc..
Application layer
Concrete application is to the organizing again and utilize one layer of data in Remote Sensing Database.For a specific application,
Its required data is generally derived from a word bank incessantly, and also partial data may be from its outside Remote Sensing Database
He originates.Application layer data is set up according to demand, its content can be tissue layer data part of records it is also possible to
Tissue layer portions data and the integration further of other derived data.
(3) remote sensing metadata standard structure
Remote sensing metadata standard structure is respectively with regard to non-relational data object, word bank, application service system and remotely-sensed data
The Metadata design in storehouse can be the standard that different data-storage systems provide unified data access interface and API specification
Data structure.
Metadata the organization and administration of data, discovery, understanding, evaluation, using etc. one or more aspects play important
Effect.In Remote Sensing Database, non-relational data object is required to have its content of announcement and relevant surface
Metadata, and the homogeneity according to comprised non-relational data object and diversity, set up one or more metadatabases
Come the metadata to preserve and to manage these non-relational data objects, (metadatabase is used for managing have some same characteristic features
, the non-relational data object that can be described with identical metadata form).And the data set for type of database
(inclusion database collection), to preserve and manages it is also desirable to set up metadatabase to support the effective management to them and discovery
Manage the metadata of these data sets.Additionally, for, the Ying Jiang integrated to Remote Sensing Database that realize market demand environment gate system
Latter metadatabase and Remote Sensing Database is registered to as entirety data center resource and service registry system
In, and the interface of compliant is opened to market demand environment gate system.
The metadata of non-relational data object
For non-relational data object, it is organized with following requirement：All of non-relational data object, must
Must there are the associated metadata elements disclosing its content characteristic, unique identifier and reference address must be comprised simultaneously；Must be set up unit
Data base non-relational data object is managed it is ensured that user can be by metadata lookup to corresponding data set；
Can be managed using same metadatabase with the non-relational data object of identical associated metadata elements collection description.
Word bank metadata
In Remote Sensing Database data service system, the effective management for the ease of word bank and discovery, a use should be set up
To preserve and to manage the metadatabase of their metadata it is desirable to its metadata must comprise system metadata and core unit number
According to may also contain field metadata in addition.System metadata mainly describes the relevant link information of word bank；Core metadata master
The substance feature of word bank to be described, surface and architectural feature, meet the regulation of core metadata specification；Field unit number
Meet the ambit metadata specification extending out based on core metadata specification according to main description.
The metadata of application service system
There is provided important data supporting for various applications, be the important content that Remote Sensing Database is built, in order to carry
The supporting level to various applications for the high thematic data base, should set up relevant application service in conjunction with the demand of concrete application
System.Relevant data in word bank required for one application has been organized in together by one application service system, and should by this
Load with required relevant model (if there is), realize the fusion of data and model.
The metadata of Remote Sensing Database
Remote Sensing Database, as an entirety, needs to register it in the resource and service Accreditation System of data center's exploitation
Metadata, and discloses in the integrated of data center's gate system in order to it.
3rd, process, towards remote sensing, the algorithm that the image procossing of cloud and high-grade products produce
The algorithm of the image procossing and high-grade products production that process cloud towards remote sensing comprises general image parallel processing algorithm
Pattern, the remote sensing image processing algorithm pattern based on cloud computing mode；
(1) general image parallel processing algorithm pattern
Although the parallel algorithm of remote sensing image processing has the characteristics that itself, it is with general data localized algorithm
The problem being related to is very similar, and simply remote sensing image processing is single file, and handling process is order execution, and therefore it is parallel
Change strategy and mainly adopt data parallel mode.The thought of image processing data parallel processing mode is more natural, is because image
Data has the characteristics that concordance and neighborhood, and this parallel schema is more suitable for cluster computing system simultaneously.Pipeline system is simultaneously
The function that each step of row completes is different, and the data of each function treatment is also different, has had function parallelization data concurrently parallel
Feature, the executed in parallel time depends on the granularity that execution time the longest step data divides, if being designed to work as, this simultaneously
Row mode can obtain very high efficiency, therefore carries out parallel computation according to handling process on the basis of data parallel in the present invention
Optimize.The transmission of task data passes through xml document with scheduling, forms standardized format, possesses cross-platform ability, be applicable to cloud
In the hind computation environment processing.
(2) the remote sensing image processing algorithm pattern based on cloud computing mode
Using the Parallel Implementation mode being combined with Map-Reduce based on MPI, using memory sharing data pre-read, right
Magnetic disc i/o operation is optimized, and can once read, and multi-process multithreading is shared, using component pattern, keep Business Logic or
" loose coupling " of business support layer function assembly, algorithm and module individual packages.
Realize general initialization module transformation function, complete efficiently autgmentability parallel schema.Provide a user with fixation
Parallel architecture, forms more easily initialization module function, and the initialization template that user passes through to provide is entered line interface and docked, and presses
Input relevant parameter according to set form, after system initialized or newly added procedure identification process, complete parallel system
Extension.
Server end running state monitoring program is adopted, by detecting to operation condition of server, to client in system
Feed back various states, reach the purpose to running state of programs real-time monitoring for the user, facilitate user to carry out next step operation management.
Based on G-Cloud control platform and data management platform, realize the premium quality product based on cloud computing node and process,
Pre-read using shared drive data, improve operation efficiency.Produce each flow of task feature for remote sensing high-grade products, will calculate
Amount is big, can the part of block parallel be improved, and improves operation efficiency；It is applied to multi-core parallel processing, the suitability is preferable；Often
The working condition of individual core and updating record, to higher management feedback, ultimately form the Mission Monitor report (xml of whole node
Form).
Remote sensing processes cloud service pattern and Demonstration Application, and for solving, user is actually used and system demonstration is verified.
Remote sensing processes cloud service pattern
Remote sensing processes service mode and adopts SOA framework, according to OGC Web service (OGC Web service, OWS) system
Build.Remote sensing image processing function in system is realized in the way of Web service and is packaged with WPS, required for service
Data all in accordance with OGC specification provide, both service chaining and execution are realized with working flow mode.Whole system is according to work(
3 levels can be divided into：Data resource service layer, function treatment service layer and application logical layer.This 3 layers design with realization is
There is provided the key of remote sensing image processing function under Web environment, the relation between them is as shown in Figure 1.
Wherein：
Data resource service layer is by OGC WCS (service of Web Raster Images), OGC WMS (web map service), OGC WFS
(Web feature services) and OGC CSW (directory service) are constituted.Wherein WCS provides Raster Images service, can return such as GeoTiff
The remote sensing image data of form；CSW provides the metadata catalog service of mass data；WFS and WMS returns vector map respectively
Data.Data resource service layer expose the interface that data source is written and read give other two-layer, data source can be file or
Data in person data base.Externally provide data resource to be the key realizing this layer in the way of meeting standard, be also remote sensing figure
As processing the data basis of workflow.
Function treatment service layer contains the various functional modules to geographical spatial data handling, and according to WPS specification pair
These modules are packed.Service in this layer is used for locating the data of reason data resource service layer offer or by user oneself
The data providing.The design of the decomposition of service and minimum service unit is also extremely important.According to the demand of different application, geographical empty
Between the complexity of information processing there is a great difference, be also much some simply dealt group to the complex process of spatial information
Close.If the WPS being directly packaged into one by one complex process services, many repeated encodings can be produced, increased programming work
While measuring, also reduce durability and the motility of service.Therefore, workflow is adopted for remote sensing image processing process
Mode is organizing.Each step in workflow is all a remote sensing images WPS service, and workflow can dynamically give birth to relative, finally
The workflow of generation is packaged as a WPS service.
Application logic control layer comprises 3 modules：Workflow storehouse, flow definition editor module and flow processing module.
Predefined flow process is comprised, for some general remote sensing image processing flow processs, these predefined flow processs in flow process storehouse
All remain algorithm interface, the specific algorithm used during user's selection can be allowed to process.
Flow definition editor module is available to the interface of User Defined handling process, flow definition editor module first with
Service register center communicates, and obtains the service list that can use, and these services are exposed to user, and user selects therein
Some Services Compositions become required handling process.After the completion of editor, user submits flow process to, and flow definition editor module generates one
XML literary composition gear, and send it to flow processing module.The workflow document of User Defined and editor can be saved in local
On device, and the XML literary composition gear being sent on server will not be remained in the way of file.
Flow processing module is responsible for process of analysis literary composition gear, original XML document is organized into the lattice meeting BPEL4WS standard
Formula, then re-sends to workflow engine, executes workflow by workflow engine.
A Web link is returned, user can obtain, by this link, the result processing after the completion of whole flow process execution,
And choose whether that the result of locally downloading process is saved in server end as temporary file, carries out next step action in user
After (preserve or close browser), interim findings are just deleted from server by flow processing module.
Demonstration Application
Demonstration Application includes just penetrating production Demonstration Application, laser radar point cloud data DEM production Demonstration Application；
Just penetrating production Demonstration Application is to process global TM base reference Image Database, the Yong Hujian of cloud management using remote sensing
Vertical control point storehouse and the 90 Miho Dockyard EM assistance datas in the whole world, call in process cloud algorithms library, are just penetrating correction and are related to algorithm and (are such as entangling
Positive model, method for resampling etc.), realize just penetrating the production of product.
In laser radar point cloud data DEM production Demonstration Application laser radar point cloud data be related to a little very intensive,
Data volume is big, will filter acquisition high quality DEM, needs complicated computing.This Demonstration Application utilizes remote sensing to process the complete of cloud management
90 Miho Dockyard EM assistance datas of ball, call process cloud algorithms library point cloud filtering algorithm, realize the production of DEM product.
Claims (6)
1. the remote sensing image processing service cloud platform system based on G-Cloud cloud computing is it is characterised in that include：
G-Cloud cloud computing platform, is used for realizing physical resource management, safety management, basic management platform and service platform, is
Remote sensing processes cloud platform and provides support, and is extended and optimizes for effectively realizing remote sensing image processing cloud platform；
Remote sensing image processing cloud platform, set up on the basis of G-Cloud cloud computing platform, include remote sensing process cloud architecture,
Process the calculation of the raster data management system, the image procossing processing cloud towards remote sensing and high-grade products production of cloud towards remote sensing
Method；
Remote sensing processes cloud service pattern and Demonstration Application, and for solving, user is actually used and system demonstration is verified.
2. the remote sensing image processing service cloud platform system based on G-Cloud cloud computing according to claim 1, its feature
It is, described remote sensing processes cloud architecture and is logically divided into host environment and resource, virtual environment, the system core, exploitation
And use environment, cloud computing remote sensing cloud service center；
Host environment and resource：Be made up of the various resources that wide area is distributed, be constitute all nodes of remote sensing cloud basic physicses and
The general name of logic entity, including the computer of node, storage system, main equipment instrument, operating system kernel, in the middle of cluster
Part；
Virtual environment：Including the platform virtualization calculating with operating system, particular system resource internal memory, storage, network system money
Source virtualizes, and application virtualization；
The system core：For connecting the systems soft ware of cloud resource；
Cloud computing remote sensing cloud service center：The various remote sensing applications of exploitation on remote sensing cloud system layer are provided, support virtualization
Remote desktop, zero-configuration client, provide software to be service function, multi-user leases multiple application software use patterns；
Cloud computing remote sensing cloud service center：The various remote sensing applications of exploitation on remote sensing cloud system layer are provided, support virtualization
Remote desktop, zero-configuration client, provide software to service (SaaS) function, multiple application software use patterns such as multi-user's lease.
3. the remote sensing image processing service cloud platform system based on G-Cloud cloud computing according to claim 1, its feature
Be, described towards remote sensing process cloud raster data management system include magnanimity remote sensing raster data cloud storage structural model,
The organizational structure of remotely-sensed data database data and remote sensing metadata standard structure；
Magnanimity remote sensing raster data cloud storage structural model：Comprise accumulation layer, basic management layer, application-interface layer and access layer,
This structural model is applied to the storage of data and the management that data scale reaches more than 100TB rank；
The organizational structure of remotely-sensed data database data：Come to complexity using these three levels of original data layer, organized layer and application layer
Remotely-sensed data carry out organizing, manage；
Remote sensing metadata standard structure：For providing unified data access interface and API rule for different data-storage systems
The standard data structure of model.
4. the remote sensing image processing service cloud platform system based on G-Cloud cloud computing according to claim 1, its feature
It is, the algorithm of the described image procossing and high-grade products production processing cloud towards remote sensing comprises general image parallel processing algorithm
Pattern, the remote sensing image processing algorithm pattern based on cloud computing mode；
In general image parallel processing algorithm pattern, the transmission of task data passes through xml document with scheduling, forms standardization lattice
Formula, possesses cross-platform ability, is applicable in the hind computation environment of cloud process；
In the remote sensing image processing algorithm pattern based on cloud computing mode, adopt and be combined simultaneously with Map-Reduce based on MPI
Row implementation, using memory sharing data pre-read, is optimized to magnetic disc i/o operation, can once read, multi-process is many
Thread is shared, and using component pattern, keeps " loose coupling ", algorithm and the module of Business Logic or business support layer function assembly
Individual packages.
5. the remote sensing image processing service cloud platform system based on G-Cloud cloud computing according to claim 1, its feature
It is, described remote sensing processes cloud service pattern and adopts SOA framework, according to OGC with remote sensing process cloud service pattern in Demonstration Application
Web service system is built, and the remote sensing image processing function in system is realized in the way of Web service and is packaged with WPS, right
All provide in accordance with the specification of OGC in the required data of service, both service chaining and execution are realized with working flow mode；
Whole system is divided by function as 3 levels：Data resource service layer, function treatment service layer and application logical layer；
Data resource service layer is made up of OGC WCS, OGC WMS, OGC WFS and OGC CSW, and wherein WCS provides Raster Images
Service, can return the remote sensing image data as GeoTiff form；CSW provides the metadata catalog service of mass data；WFS and
WMS returns vector map datum respectively；
Function treatment service layer contains the various functional modules to geographical spatial data handling, and according to WPS specification to these
Module is packed；
Application logic control layer comprises 3 modules：Workflow storehouse, flow definition editor module and flow processing module.
6. the remote sensing image processing service cloud platform system based on G-Cloud cloud computing according to claim 1, its feature
It is, described remote sensing processes cloud service pattern and includes just penetrating production Demonstration Application, laser with Demonstration Application in Demonstration Application
Radar cloud data DEM production Demonstration Application；
Just penetrating production Demonstration Application is to process global TM base reference Image Database, user's foundation of cloud management using remote sensing
Control point storehouse and the 90 Miho Dockyard EM assistance datas in the whole world, call in process cloud algorithms library, are just penetrating correction and are being related to algorithm, realize just penetrating
The production of product；
In laser radar point cloud data DEM production Demonstration Application, laser radar point cloud data is related to a little very intensive, data
Amount is big, will filter acquisition high quality DEM, needs complicated computing, and this Demonstration Application utilizes remote sensing to process the whole world of cloud management
90 Miho Dockyard EM assistance datas, call process cloud algorithms library point cloud filtering algorithm, realize the production of DEM product.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611005463.2A CN106453618A (en) | 2016-11-15 | 2016-11-15 | Remote sensing image processing service cloud platform system based on G-Cloud cloud computing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611005463.2A CN106453618A (en) | 2016-11-15 | 2016-11-15 | Remote sensing image processing service cloud platform system based on G-Cloud cloud computing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106453618A true CN106453618A (en) | 2017-02-22 |
Family
ID=58207172
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611005463.2A Pending CN106453618A (en) | 2016-11-15 | 2016-11-15 | Remote sensing image processing service cloud platform system based on G-Cloud cloud computing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106453618A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107273867A (en) * | 2017-06-27 | 2017-10-20 | 航天星图科技（北京）有限公司 | Empty day Remote Sensing Data Processing all-in-one |
| CN107423053A (en) * | 2017-06-15 | 2017-12-01 | 东莞理工学院 | The webization model encapsulation and distributed approach of a kind of remote sensing image processing |
| CN108091384A (en) * | 2017-11-09 | 2018-05-29 | 南京耐久软件科技有限公司 | A kind of medical image processing system based on micro services frame |
| CN108234612A (en) * | 2017-12-23 | 2018-06-29 | 北京卫星信息工程研究所 | The high score Resource dynamic allocation of multi-terminal mode and on-demand service delivery system |
| CN109218969A (en) * | 2018-10-19 | 2019-01-15 | 程桂平 | Enhance the method for the image-capable of cloud computing platform |
| CN110059122A (en) * | 2019-04-03 | 2019-07-26 | 广州地理研究所 | A kind of wildlife resource data storage method based on cloud Web |
| CN111241223A (en) * | 2020-01-07 | 2020-06-05 | 中化现代农业有限公司 | Remote sensing automatic processing system and remote sensing automatic processing method |
| CN112115198A (en) * | 2020-09-14 | 2020-12-22 | 宁波市测绘和遥感技术研究院 | Urban remote sensing intelligent service platform |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101546325A (en) * | 2008-12-23 | 2009-09-30 | 重庆邮电大学 | Grid heterogeneous data integrating method based on SOA |
| CN101945126A (en) * | 2010-09-09 | 2011-01-12 | 中国林业科学研究院资源信息研究所 | Forest resource heterogeneous data distributed management system |
| CN102904944A (en) * | 2012-09-29 | 2013-01-30 | 广东中科遥感技术有限公司 | Remote cloud processing system and method for remote sensing data |
| CN103002014A (en) * | 2012-11-09 | 2013-03-27 | 哈尔滨中智拓图地理信息技术有限公司 | Environmental geographic information service platform based on cloud computing and internet-of-things technology |
| CN103019671A (en) * | 2012-10-08 | 2013-04-03 | 中国科学院对地观测与数字地球科学中心 | Data-intensive remote sensing image processing-oriented generic programmed framework programming method |
| CN103259872A (en) * | 2013-05-31 | 2013-08-21 | 江苏物联网研究发展中心 | Multi-source heterogeneous geographic information service platform based on open-type grid system |
| CN103281351A (en) * | 2013-04-19 | 2013-09-04 | 武汉方寸科技有限公司 | Cloud server platform capable of efficiently processing and analyzing remote sensing data |
| CN103595791A (en) * | 2013-11-14 | 2014-02-19 | 中国科学院深圳先进技术研究院 | Cloud accessing method for mass remote sensing data |
| CN104506632A (en) * | 2014-12-25 | 2015-04-08 | 中国科学院电子学研究所 | Resource sharing system and method based on distributed multi-center |
| CN104794150A (en) * | 2015-01-30 | 2015-07-22 | 北京东方泰坦科技股份有限公司 | Cloud storage model and management method based on space knowledge cloud environment |
| CN105487892A (en) * | 2015-11-26 | 2016-04-13 | 武大吉奥信息技术有限公司 | Inter-cloud GIS service deploying system in Linux environment |
-
2016
- 2016-11-15 CN CN201611005463.2A patent/CN106453618A/en active Pending
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101546325A (en) * | 2008-12-23 | 2009-09-30 | 重庆邮电大学 | Grid heterogeneous data integrating method based on SOA |
| CN101945126A (en) * | 2010-09-09 | 2011-01-12 | 中国林业科学研究院资源信息研究所 | Forest resource heterogeneous data distributed management system |
| CN102904944A (en) * | 2012-09-29 | 2013-01-30 | 广东中科遥感技术有限公司 | Remote cloud processing system and method for remote sensing data |
| CN103019671A (en) * | 2012-10-08 | 2013-04-03 | 中国科学院对地观测与数字地球科学中心 | Data-intensive remote sensing image processing-oriented generic programmed framework programming method |
| CN103002014A (en) * | 2012-11-09 | 2013-03-27 | 哈尔滨中智拓图地理信息技术有限公司 | Environmental geographic information service platform based on cloud computing and internet-of-things technology |
| CN103281351A (en) * | 2013-04-19 | 2013-09-04 | 武汉方寸科技有限公司 | Cloud server platform capable of efficiently processing and analyzing remote sensing data |
| CN103259872A (en) * | 2013-05-31 | 2013-08-21 | 江苏物联网研究发展中心 | Multi-source heterogeneous geographic information service platform based on open-type grid system |
| CN103595791A (en) * | 2013-11-14 | 2014-02-19 | 中国科学院深圳先进技术研究院 | Cloud accessing method for mass remote sensing data |
| CN104506632A (en) * | 2014-12-25 | 2015-04-08 | 中国科学院电子学研究所 | Resource sharing system and method based on distributed multi-center |
| CN104794150A (en) * | 2015-01-30 | 2015-07-22 | 北京东方泰坦科技股份有限公司 | Cloud storage model and management method based on space knowledge cloud environment |
| CN105487892A (en) * | 2015-11-26 | 2016-04-13 | 武大吉奥信息技术有限公司 | Inter-cloud GIS service deploying system in Linux environment |
Non-Patent Citations (2)
| Title |
|---|
| 曾志: "云格环境下海量高分遥感影像资源与服务高效调配研究", 《万方全文数据库》 * |
| 李春雷: "基于云平台的遥感业务流程研究及原型实现", 《中国优秀硕士论文全文数据库》 * |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107423053B (en) * | 2017-06-15 | 2020-08-21 | 东莞理工学院 | Web model packaging and distributed processing method for remote sensing image processing |
| CN107423053A (en) * | 2017-06-15 | 2017-12-01 | 东莞理工学院 | The webization model encapsulation and distributed approach of a kind of remote sensing image processing |
| CN107273867A (en) * | 2017-06-27 | 2017-10-20 | 航天星图科技（北京）有限公司 | Empty day Remote Sensing Data Processing all-in-one |
| CN108091384A (en) * | 2017-11-09 | 2018-05-29 | 南京耐久软件科技有限公司 | A kind of medical image processing system based on micro services frame |
| CN108091384B (en) * | 2017-11-09 | 2021-07-09 | 南京耐久软件科技有限公司 | Medical image processing system based on micro-service framework |
| CN108234612B (en) * | 2017-12-23 | 2021-07-06 | 北京卫星信息工程研究所 | Multi-terminal mode high-resolution resource dynamic allocation and on-demand service release system |
| CN108234612A (en) * | 2017-12-23 | 2018-06-29 | 北京卫星信息工程研究所 | The high score Resource dynamic allocation of multi-terminal mode and on-demand service delivery system |
| CN109218969A (en) * | 2018-10-19 | 2019-01-15 | 程桂平 | Enhance the method for the image-capable of cloud computing platform |
| CN110059122A (en) * | 2019-04-03 | 2019-07-26 | 广州地理研究所 | A kind of wildlife resource data storage method based on cloud Web |
| CN110059122B (en) * | 2019-04-03 | 2022-03-18 | 广州地理研究所 | Wild animal resource data storage method based on cloud Web |
| CN111241223A (en) * | 2020-01-07 | 2020-06-05 | 中化现代农业有限公司 | Remote sensing automatic processing system and remote sensing automatic processing method |
| CN112115198A (en) * | 2020-09-14 | 2020-12-22 | 宁波市测绘和遥感技术研究院 | Urban remote sensing intelligent service platform |
| CN112115198B (en) * | 2020-09-14 | 2024-03-15 | 宁波市测绘和遥感技术研究院 | Urban remote sensing intelligent service platform |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106453618A (en) | Remote sensing image processing service cloud platform system based on G-Cloud cloud computing | |
| US11150960B2 (en) | Distributed application allocation and communication | |
| CN105978704B (en) | System and method for creating new cloud resource instruction set architecture | |
| CN104253865B (en) | A kind of two-level management method of mixed type desktop cloud service platform | |
| US9465589B2 (en) | Stateful component authoring and execution | |
| CN105830049B (en) | Automation experiment platform | |
| US9081837B2 (en) | Scoped database connections | |
| CN103246749B (en) | The matrix database system and its querying method that Based on Distributed calculates | |
| CN108885582A (en) | The multi-tenant memory services of memory pool structure | |
| CN107766402A (en) | A kind of building dictionary cloud source of houses big data platform | |
| CN109643312A (en) | Trustship query service | |
| US20120254109A1 (en) | Distributed component runtime | |
| CN104008135A (en) | Multi-source heterogeneous database fusion system and data query method thereof | |
| CN107315776A (en) | A kind of data management system based on cloud computing | |
| US11416553B2 (en) | Spatial indexing | |
| Essa et al. | Mobile agent based new framework for improving big data analysis | |
| Wang et al. | Research and implementation on spatial data storage and operation based on Hadoop platform | |
| US11436217B2 (en) | Ordered append-only log based data storage | |
| CN106709048A (en) | High-performance computing-oriented distributed data organization method | |
| CN104820946A (en) | Cloud computing system for agricultural information integration | |
| Wang et al. | Retrieving and indexing spatial data in the cloud computing environment | |
| CN103595799A (en) | Method for achieving distributed shared data bank | |
| CN104281980B (en) | Thermal power generation unit remote diagnosis method and system based on Distributed Calculation | |
| CN108268614A (en) | A kind of distribution management method of forest reserves spatial data | |
| CN109063980A (en) | Memory calculation method and system suitable for electrical network analysis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination |