CN112351079B - Space application and data integrated packaging system and method based on data box - Google Patents
Space application and data integrated packaging system and method based on data box Download PDFInfo
- Publication number
- CN112351079B CN112351079B CN202011147628.6A CN202011147628A CN112351079B CN 112351079 B CN112351079 B CN 112351079B CN 202011147628 A CN202011147628 A CN 202011147628A CN 112351079 B CN112351079 B CN 112351079B
- Authority
- CN
- China
- Prior art keywords
- data
- box
- data box
- application
- service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
- H04L67/025—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP] for remote control or remote monitoring of applications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
- G06F8/61—Installation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
- G06F8/61—Installation
- G06F8/62—Uninstallation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/70—Software maintenance or management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/2866—Architectures; Arrangements
- H04L67/30—Profiles
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/34—Network arrangements or protocols for supporting network services or applications involving the movement of software or configuration parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/51—Discovery or management thereof, e.g. service location protocol [SLP] or web services
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45562—Creating, deleting, cloning virtual machine instances
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45575—Starting, stopping, suspending or resuming virtual machine instances
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45591—Monitoring or debugging support
Abstract
The invention belongs to the technical field of spatial information data integration, and discloses a system and a method for integrally packaging spatial application and data based on data boxes, wherein the method comprises the steps of arranging a data box operation and maintenance subsystem in each data box, and carrying out application management and container management on the data boxes through the data box operation and maintenance subsystem; performing attribute configuration on the data box through the data box operation and maintenance subsystem; the data box is canned and published by the data box operation and maintenance subsystem; the data boxes are integrated in a cloud mode through the cooperation of the data box operation and maintenance subsystem and the data honeycomb management subsystem; and dynamically scheduling the data boxes through the data honeycomb management subsystem. The system comprises a data box operation and maintenance subsystem and a data honeycomb management subsystem. The invention solves the problem of huge workload of manual operation of spatial information data integration in the prior art, can effectively save a large amount of manpower and material resources cost spent in integrating data and application, and realizes the pluggable automatic integration of data application integration.
Description
Technical Field
The invention relates to the technical field of spatial information data integration, in particular to a data box-based spatial application and data integrated packaging system and method.
Background
In the construction of a spatial information cloud subsystem, massive heterogeneous data from various data providers such as a satellite ground receiving station, a meteorological department, an unmanned aerial vehicle monitoring station, a traffic department and the like needs to be collected, and various spatial applications aiming at processing and analysis of specific data need to be integrated. Therefore, it is a difficult task to integrate data and applications, and these tasks usually require a lot of manpower and material resources to complete.
At present, the spatial data access mainly includes metadata access, data volume access, data service access and other contents. Metadata access is usually provided by a data provider for metadata description files, extracted and converted, or directly provided with an online query interface for automatic capture by external applications. The access of the data body adopts FTP to upload files, adopts carriers such as optical disks to copy files, and provides network addresses for the integration party to download remotely. Data service access is realized by directly providing a service address by a data provider or manually releasing service.
For example, a distributed system architecture is adopted in an american EOS (earth observing system), remote sensing data is divided into various data centers according to the scientific field for storage, and a unified data storage format (i.e., HDF-EOS data format) is adopted as a standard format of an EOS data product, and a unified data access interface is provided. In the aspect of remote sensing image distribution, the EOSDIS adopts an FTP mode, when a large number of users make data requests, the load of an FTP server is increased rapidly, the bandwidth performance of the server is insufficient, and high-quality and rapid data distribution service cannot be provided for the users. The Google Earth Engine is a cloud platform provided by Google for online visual computation and analysis processing of a large amount of global-scale geoscience data (particularly satellite data). The platform has access to data in satellite images and other earth observation data databases and provides sufficient computing power to process such data. The Google Earth Engine provides not only an online JavaScript API, but also an offline Python API. Web services based on Google Earth Engine and Google cloud can be established rapidly through the APIs.
However, when the spatial data is put into a database in the metadata access mode, the above system needs much customized development work, such as writing codes, implementing entry and conversion of metadata, and implementing collection, indexing and query of metadata. The data body can be used only by corresponding processing, for example, relational data needs to be manually or encoded and put in storage, vector data needs to be published as corresponding data service, data needing to be visually displayed needs to be sliced, and data service needs to be deployed and published after being accessed. Therefore, space application and data integration need to package heterogeneous applications into cloud services by compiling codes, the work of packaging, integration, deployment, configuration, testing and the like of the services needs to be completed manually, high availability and elastic expansion and contraction need to be achieved by the user, and the workload of manual operation is huge.
Disclosure of Invention
The invention provides a space application and data integrated packaging system and method based on a data box, and solves the problem of huge workload of manual operation of space information data integration in the prior art.
The invention provides a space application and data integrated packaging method based on a data box, which comprises the following steps:
step 1, a data box operation and maintenance subsystem is arranged in each data box, and application management and container management of the data boxes are performed through the data box operation and maintenance subsystem;
step 2, performing attribute configuration on the data box through the data box operation and maintenance subsystem;
step 3, canning and publishing the data box through the data box operation and maintenance subsystem;
step 4, performing cloud integration on the data box through the cooperation of the data box operation and maintenance subsystem and the data honeycomb management subsystem;
and 5, dynamically scheduling the data box through the data honeycomb management subsystem.
Preferably, in step 1, the application management includes installing an application, uninstalling the application, querying and modifying basic information of the application in batch, and querying and modifying built-in service of the application;
the container management comprises starting the container, restarting the container, stopping the container, inquiring the container state, viewing the container log, searching the container, viewing and modifying the container data volume.
Preferably, in the step 2, the attribute configuration includes a custom configuration, and a cooperative configuration of a data box and a data cell; the custom configuration includes, but is not limited to, a metadata conversion rule configuration, a cell address configuration.
Preferably, in the step 3, the canning publishing of the data box includes the following sub-steps:
pre-loading an application, selecting a container mirror image, and creating a group of containers;
canning data, canning data and metadata thereof in a container;
packaging and releasing, mirroring the canned data, the metadata and the container, and packaging into a data box mirror image.
Preferably, in the step 3, the metadata is imported into an ElasticSearch; the data comprises file data and structured data; the file data comprises an image file and a vector file; storing the image file and the vector file in a Minio Server in an objectification manner; inputting the vector file into a GEOServer to release OGC data service; and inputting the structured data into MongoDB for storage.
Preferably, in the step 4, the cloud integration of the data boxes includes: data box service discovery and registration;
the data box operation and maintenance subsystem is used as a Eureka Client, and the data honeycomb management subsystem is used as a Eureka Server; after application management starts an application program, the Eureka Client registers the information of a service node to the Eureka Server and issues the application as a service; inquiring the information of the service node through the Eureka Server; and the Eureka Client sends a heartbeat to the Eureka Server, and if the Eureka Server does not receive the heartbeat of a certain service node within a preset heartbeat period, the service node is removed from the service registry.
Preferably, in the step 4, the cloud integration of the data boxes includes: automatically converging metadata of the data boxes;
after the data box is started, the data box operation and maintenance subsystem arranged in the data box communicates with the data honeycomb management subsystem, and the data box automatically registers metadata and application service to the data honeycomb management subsystem.
Preferably, in the step 4, the cloud integration of the data boxes includes: managing the life cycle of the data box;
the cloud end is used for retrieving and downloading the data and the metadata of the data in the data boxes in a distributed mode, the cloud end is used for processing and analyzing the data and the metadata of the data in the data boxes on line, and the life cycle management of the data boxes is achieved.
Preferably, in the step 5, the dynamically scheduling the data box includes:
the autonomous application uniform access uses the service gateway Zuul to dynamically configure the routing of the proxy according to the issued service information, and the reverse proxy function is realized;
autonomous application cluster access, and automatic load balancing of a plurality of data box services is realized by integrating Ribbon in a service gateway Zuul;
and dynamically updating the micro service nodes, namely dynamically updating the proxy mapping and the cluster nodes when the service gateway Zuul is used for online and offline of the data box, so as to realize the function of dynamically updating the nodes.
In another aspect, the present invention provides a data box-based space application and data integration packaging system, including: the data box operation and maintenance subsystem and the data honeycomb management subsystem;
the space application and data integration packaging system based on the data box is used for realizing the steps in the space application and data integration packaging method based on the data box.
One or more technical schemes provided by the invention at least have the following technical effects or advantages:
in the invention, each data box is internally provided with a data box operation and maintenance subsystem, and the application management and the container management of the data box are carried out through the data box operation and maintenance subsystem; performing attribute configuration on the data box through the data box operation and maintenance subsystem; the data box is canned and published by the data box operation and maintenance subsystem; the data boxes are integrated in a cloud mode through the cooperation of the data box operation and maintenance subsystem and the data honeycomb management subsystem; and dynamically scheduling the data boxes through the data honeycomb management subsystem. According to the invention, through the steps of application and container management, data box attribute configuration, data box filling box release, data box clouding integration, data box dynamic scheduling and the like, data dynamic access and application clouding integration are completed, a large amount of manpower and material resource cost spent in data and application integration can be effectively saved, and the pluggable automatic integration of data application integration is realized.
Drawings
Fig. 1 is a swim lane flowchart corresponding to a data box-based space application and data integration packaging method according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a data box canning process in a space application and data integrated packaging method based on a data box according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a data box clouding integration process in a space application and data integration packaging method based on a data box according to an embodiment of the present invention;
fig. 4 is a schematic diagram of a service reverse proxy process in a space application and data integration encapsulation method based on a data box according to an embodiment of the present invention.
Detailed Description
In order to better understand the technical solution, the technical solution will be described in detail with reference to the drawings and the specific embodiments.
The embodiment provides a space application and data integrated packaging method based on a data box, and with reference to fig. 1, the method comprises the following steps:
step 1, a data box operation and maintenance subsystem is arranged in each data box, and application management and container management of the data boxes are performed through the data box operation and maintenance subsystem;
step 2, performing attribute configuration on the data box through the data box operation and maintenance subsystem;
step 3, canning and publishing the data box through the data box operation and maintenance subsystem;
step 4, performing cloud integration on the data box through the cooperation of the data box operation and maintenance subsystem and the data honeycomb management subsystem;
and 5, dynamically scheduling the data box through the data honeycomb management subsystem.
The present invention is further described below.
The embodiment provides a space application and data integrated packaging method based on a data box, which comprises the following steps:
step 1, application and container management.
An operation and maintenance subsystem is arranged in each data box, and a data box holder manages the application and the container of the data box through the operation and maintenance subsystem.
The application management functions include: installing an application, uninstalling the application, inquiring and modifying basic application information in batch, and inquiring and modifying built-in application services.
The container management functions include: starting the container, restarting the container, stopping the container, inquiring the container state, checking the container log, searching the container, checking and modifying the container data volume.
And 2, configuring the attribute of the data box.
And self-defined configurations such as metadata conversion rules, honeycomb addresses and the like are supported, so that the individual customization requirements of different users on the data box are met. Meanwhile, the cooperative configuration of the data box operation and maintenance subsystem and the data honeycomb management subsystem is supported.
The data box is an autonomous running software entity formed by integrally packaging data and related applications. The data box can run independently, provides various related services such as data box metadata query and access, and achieves the purpose of 'using after distribution'. Meanwhile, the data box automatically integrates data, metadata and application in the data box through a management platform named as a data honeycomb, so that pluggable automatic integration is realized, and the workload of space data integration is greatly reduced.
Step 3, canning and publishing the data box:
the data box canning publishing process is shown in fig. 2. The data box canning release comprises three steps: pre-installing the application, canning the data and packaging and publishing. And in the pre-installation application stage, a container mirror image is selected, and a group of containers is created. And a data canning stage, namely canning data, metadata and publishing data services in a container. And in the packaging and publishing stage, the canned data, the metadata and the container are mirrored and packaged into a data box mirror image for deployment.
Step 3.1: the application is preloaded.
In a data box, autonomous applications run in containers that are isolated from each other. The autonomous applications in the container comprise metadata management, object storage, data service release, data slicing, structured data storage and access and the like, and also comprise two autonomous applications of operation management and safety management of the data box.
A Docker container is created using mirroring. Through the command line, the user can conveniently control the operations of creating, starting, stopping, deleting and pausing the container. The pre-load application is implemented by selecting an existing container image and creating the corresponding container.
Each container is a subsystem which is isolated from each other and can ensure safety. The container can be regarded as a simple version of Linux environment (including root user rights, process space, user space, and network space, etc.) and the applications running therein.
According to Docker's best practice, data files are read and written at these locations using a way that the data volumes bind the hosting directories. This may lead to better performance and higher stability. Because the data volume exists independently of the container, the operation on the container does not affect the data in the data volume.
In different applications, the data to be canned has a data volume, the data volume is mapped to a certain directory of the disk, and then when the data is canned subsequently, the generated data is stored under the corresponding directory.
Step 3.2: and (6) canning the data.
When the container creation is successful, the next step is to pack the data into the container. The data and metadata to be canned are stored in a file form as input for canning.
And (3) importing metadata: the metadata describes each spatial data using an XML document form. Raw metadata documents and XSLT documents are imported as input into the library of the ElasticSearch to build an index to support the query and access of metadata.
Storing objects of image files and vector files: the file data of the video file and the vector file are managed by means of object storage. And storing the file data into a storage facility for the canning tool through a storage interface of the Minio Server, and accessing the file data through the marked cloud storage interface (S3 interface).
And (3) publishing of the data service: for the vector file, if the vector file needs to be published as services of WMS, WCS and the like of OGC specification, the vector file can be published through a built-in data box operation and maintenance subsystem, and the published vector file can support a service mode to be accessed.
And (3) warehousing of structured data: for structured data, a distributed database MongoDB is adopted in a data box for storage, and the MongoDB can support various data storage and has extremely high storage and query efficiency. Therefore, the structured data is directly stored in the MongoDB, and a Rest-based structured data access interface is provided for inquiring and accessing the data.
Step 3.3: and (5) packaging and releasing.
After the data canning is completed, various heterogeneous data and metadata are stored in the data volume of the container, i.e., below the corresponding folders. And integrally packaging the mirror image files of each autonomous application and the folders corresponding to the data volumes to generate a data box mirror image which can be allowed to be deployed. The image may be further deployed to operate in a data cell.
The mirror image of the data box can be transmitted to other cloud subsystems (data centers) for integration through network transmission or can be copied through an offline mobile storage device.
And 4, step 4: and (4) cloud integration of the data boxes.
The data box cloud integration is based on the cooperation of the data honeycomb management subsystem and the data box operation and maintenance subsystem, and realizes the automatic discovery and collection of metadata in the data box, the automatic proxy of service and the virtual integration of data. The method specifically comprises the following steps: data box service discovery and registration, data box metadata automatic aggregation, and data box life cycle management.
After the data box is started, the data box operation and maintenance subsystem in the data box communicates with the data honeycomb management subsystem to implement pluggable cloud integration, as shown in fig. 3. The data box operation and maintenance subsystem can automatically register metadata with the data honeycomb management subsystem to realize automatic collection of data; meanwhile, through a service discovery mechanism, each autonomous application in the form of micro-service is automatically discovered by a service discovery engine in the data honeycomb management subsystem and registered.
Step 4.1: data box service discovery and registration.
For the application integration requirement of the data honeycomb and the data box, in the data honeycomb management subsystem, a Eureka service discovery engine is used for realizing the discovery of the data box. The method comprises the steps that the Eureka Server is used for providing a service registration function, and after each service node integrating the Eureka Client is started, information of the service is registered in the Eureka Server, wherein the information comprises a host and a port number, a service version number, a communication protocol and the like. The Eureka Server provides a visual web interface to view the information of these service nodes. Meanwhile, the Eureka Server is used for supporting the deployment of the cluster mode, the Eureka Server is connected end to form a closed loop, different service registration centers in the cluster mutually copy respective states through an asynchronous mode, and this also means that state inconsistency of services in various instances is completely possible to occur at specific same time nodes.
To solve the above problem, after the application management starts the application, the Eureka Client will send a heartbeat (the default period is 30 seconds) to the Eureka Server in addition to registering the service provided by itself with the Eureka Server. If the Eureka Server does not receive a heartbeat for a node over multiple heartbeat cycles, the Eureka Server will remove the service node from the service registry (default 90 seconds). At the same time, he can also query the currently registered service information from the service end and cache them locally and periodically refresh the service state of the line.
Step 4.2: the data box metadata is automatically collected.
The data boxes are canned based on the Docker technology. One data box mirror image comprises a plurality of Docker mirror images which respectively correspond to a plurality of autonomous applications in the box, and simultaneously comprises data corresponding to the applications. After deployment and startup are performed in the data box cloud, the multiple Docker images all create containers and are started in the cloud.
When the data box is started, its status is discovered by the client in real time and can communicate with the data cell management subsystem. The data box can automatically register metadata with the data honeycomb management subsystem to realize automatic collection of data; meanwhile, through a service discovery mechanism, each autonomous application in the form of micro-service is automatically discovered by a service discovery engine in the data honeycomb management subsystem and registered. And when the data box is off-line, the data honeycomb management subsystem can find the data box in real time, timely strip corresponding metadata and perform de-registration on the autonomous application.
Meanwhile, aiming at the data integration requirement of the data honeycomb management subsystem and the data box, an automatic metadata collection technology is adopted to realize the integration of metadata. After the data boxes are online, the data honeycomb management subsystem is communicated with the data box operation and maintenance subsystem to collect metadata of the distributed dynamically accessed data boxes, and a global index of the data is constructed through data association mapping, data clustering and metadata standard conversion. And the data retrieval can be integrated based on the global index, and the data scattered in each data box can be retrieved. The data access proxy problem in the data box is realized by using the service gateway, the object storage service and the relational database in the data box are registered in the service gateway, and when the external requests access the data, the automatic forwarding is realized.
Step 4.3: and managing the life cycle of the data box.
The data honeycomb management subsystem carries out virtualized organization and automatic integration on a plurality of data boxes issued by the cloud end, and unified metadata management and distributed data processing are achieved. In the data cell management subsystem, a plurality of data boxes may be started, the started data boxes may be automatically discovered by the data cell management subsystem, metadata in the data boxes may be automatically registered in a metadata management center of the data cell management subsystem, and application services in the data boxes may also be registered in a service management center of the data cell management subsystem.
The user can retrieve the data in the data boxes and the metadata thereof at the cloud end, and provide distributed downloading of the data through the data boxes. Data in the data box and metadata processing and query services of the data box can be called by a data production processor, cloud online processing and analysis of the data are achieved, and life cycle management of the data box is achieved.
And 5: and dynamically scheduling the data boxes.
And data box dynamic scheduling, including unified access and cluster access of autonomous applications. Wherein, the reverse proxy technology based on the service gateway can uniformly access all autonomous applications in the data honeycomb management subsystem through the service gateway. Meanwhile, based on the cluster scheduling function provided by the service gateway, the cluster access of the same data box mirror image autonomous application is realized. In addition, the reliability of data access is guaranteed by dynamically updating the micro service nodes.
Step 5.1: and unified access is performed on the autonomous applications.
The reverse proxy function is implemented using the service gateway Zuul as shown in fig. 4. The service gateway can obtain services registered in the Eureka in the service registration center, and the service list in the Eureka server can be dynamically changed according to the starting and stopping of the micro-service. Thus, when a request for service invocation (data access) is made from the outside, the request is dynamically transferred to the corresponding intranet service. With the service discovery engine, Zuuul can automatically configure the routing of the agent according to the service information in Eureka, and implement the service agent in zero configuration.
Step 5.2: autonomous application cluster access.
In order to realize automatic load balancing of a plurality of data box services, a Ribbon is integrated in a service gateway. The Ribbon is a component for realizing load balancing processing by a client. When there are more external requests, the same data box can be mirrored to start multiple data box instances. Multiple instances of the data box are found in the honeycomb and automatically registered in the service gateway after being discovered by the Eureka, so as to form a cluster. When external requests are made, the service gateway automatically selects a service of one data box instance in the cluster to call according to the load balancing test, and a plurality of calls are distributed to a plurality of data box instances, so that load balancing is realized.
Step 5.3: and the micro service node is dynamically updated.
The service gateway provides a node dynamic updating function, and dynamically updates the proxy mapping and the cluster nodes when the data box is online or offline, so that the reliability of data access is guaranteed.
In another aspect, an embodiment of the present invention provides a data-box-based space application and data integration packaging system, which includes a data-box operation and maintenance subsystem and a data-cell management subsystem.
The data box operation and maintenance subsystem packages data and micro-servitized applications into 'data boxes' which can be independently deployed and run autonomously.
The data honeycomb management subsystem is used for automatically discovering and registering, automatically collecting metadata, automatically realizing a center of dynamic scheduling of a data box, finishing dynamic data access and application cloud integration, and realizing pluggable automatic integration of data application integration.
Specifically, the space application and data integration packaging system based on the data box is used for realizing the steps in the space application and data integration packaging method based on the data box.
In summary, the invention completes the dynamic data access and the application clouded integration by the steps of application and container management, data box attribute configuration, data box filling box release, data box clouded integration, data box dynamic scheduling and the like, can effectively save a large amount of manpower and material resources cost spent in integrating data and application, and realizes the pluggable automatic integration of data application integration.
The invention can provide the following for the user:
(1) the data box is an autonomous running software entity formed by integrally packaging data and related applications. The data box can run independently, provides various related services such as data box metadata query and access, and achieves the purpose of 'using after distribution'.
(2) Data, metadata and application in the data box can be automatically integrated, pluggable automatic integration is realized, and the workload of space data integration is greatly reduced.
(3) Supporting automated processing, transformation and automated organization, management and access control of metadata, data volumes and data services
(4) And the automation, multiple aspects and deep level expansion of the data box are supported.
The space application and data integrated packaging system and method based on the data box provided by the embodiment of the invention at least comprise the following technical effects:
(1) plug-in cloud integration can be realized. Based on the direct cooperation of the data box operation and maintenance subsystem and the data honeycomb management subsystem, the automatic discovery of the data box, the automatic collection of metadata, the automatic proxy of services in the data box and the virtual integration of data in the data box can be realized. And can be realized by a plurality of data boxes, and the load balance of the cluster operation boxes is needed. After the data box is offline, the offline state can be automatically detected, and pluggable cloud integration capability is provided.
(2) Automated metadata transformation and management. With built-in applications, the data box can support querying, updating, and transformation of metadata. Metadata retrieval service is built in the data box to realize various data retrieval; a built-in metadata updating service for updating metadata at regular time; meanwhile, metadata conversion service is built in, and automatic metadata conversion is realized by customizing a mapping rule.
(3) Controlled distribution of data. By arranging tracking, charging, quota, encryption and access control applications in the data box, the data in the data box can be prevented from being illegally used by external users, so that a data provider can reasonably and effectively control the distribution of the data.
(4) On-demand distribution of data. By means of the on-demand pushing mechanism of the data built in the data boxes, when a user requests a data body, the data body can be synchronously distributed to the nearest data box on demand, and unnecessary data space occupation is reduced.
(5) Online distribution of applications. By embedding the application market client and the function of installing the application in the data box, a user of the data box can download the application from the application market online and install the application in the data box, so that the online distribution of the application is realized.
Finally, it should be noted that the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention has been described in detail with reference to examples, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, which should be covered by the claims of the present invention.
Claims (6)
1. A space application and data integrated packaging method based on a data box is characterized by comprising the following steps:
step 1, a data box operation and maintenance subsystem is arranged in each data box, and application management and container management of the data boxes are performed through the data box operation and maintenance subsystem;
step 2, performing attribute configuration on the data box through the data box operation and maintenance subsystem;
step 3, canning and publishing the data box through the data box operation and maintenance subsystem;
in the step 3, the canning publishing of the data box comprises the following substeps:
pre-loading an application, selecting a container mirror image, and creating a group of containers;
canning data, canning data and metadata thereof in a container;
packaging and releasing, namely mirroring the canned data, the metadata and the container and packaging the mirrored data, the metadata and the container into a data box mirror image;
step 4, performing cloud integration on the data box through the cooperation of the data box operation and maintenance subsystem and the data honeycomb management subsystem;
in step 4, the cloud integration of the data boxes includes: data box service discovery and registration, data box metadata automatic aggregation and data box life cycle management;
the data box operation and maintenance subsystem is used as a Eureka Client, and the data honeycomb management subsystem is used as a Eureka Server; after application management starts an application program, the Eureka Client registers the information of a service node to the Eureka Server and issues the application as a service; inquiring the information of the service node through the Eureka Server; the method comprises the steps that a heartbeat is sent to a Eureka Server by the Eureka Client, and if the Eureka Server does not receive the heartbeat of a certain service node in a preset heartbeat period, the service node is removed from a service registry;
after the data box is started, the data box operation and maintenance subsystem arranged in the data box communicates with the data honeycomb management subsystem, and the data box automatically registers metadata and application service to the data honeycomb management subsystem;
the cloud end is utilized to retrieve and download the data and the metadata of the data in the data boxes in a distributed manner, and the cloud end is utilized to perform online processing and analysis on the data and the metadata of the data in the data boxes, so that the life cycle management of the data boxes is realized;
and 5, dynamically scheduling the data box through the data honeycomb management subsystem.
2. The integrated packaging method for space application and data based on data box according to claim 1, wherein in the step 1, the application management comprises installing application, uninstalling application, querying and modifying application basic information in batch, querying and modifying application built-in service;
the container management comprises starting the container, restarting the container, stopping the container, inquiring the container state, viewing the container log, searching the container, viewing and modifying the container data volume.
3. The method for integrally encapsulating data and space based on data box according to claim 1, wherein in the step 2, the attribute configuration comprises a custom configuration, a cooperative configuration of data box and data cell; the custom configuration includes, but is not limited to, a metadata conversion rule configuration, a cell address configuration.
4. The data box-based space application and data integration packaging method according to claim 1, wherein in the step 3, the metadata is imported into an elastic search; the data comprises file data and structured data; the file data comprises an image file and a vector file; storing the image file and the vector file in a Minio Server in an objectification manner; inputting the vector file into a GEOServer to release OGC data service; and inputting the structured data into MongoDB for storage.
5. The method for integrally encapsulating data and space based on data boxes according to claim 1, wherein in the step 5, the dynamically scheduling the data boxes includes:
the autonomous application uniform access uses the service gateway Zuul to dynamically configure the routing of the proxy according to the issued service information, and the reverse proxy function is realized;
autonomous application cluster access, and automatic load balancing of a plurality of data box services is realized by integrating Ribbon in a service gateway Zuul;
and dynamically updating the micro service nodes, namely dynamically updating the proxy mapping and the cluster nodes when the service gateway Zuul is used for online and offline of the data box, so as to realize the function of dynamically updating the nodes.
6. A data box-based space application and data integrated packaging system is characterized by comprising: the data box operation and maintenance subsystem and the data honeycomb management subsystem;
the data box-based space application and data integration packaging system is used for realizing the steps in the data box-based space application and data integration packaging method according to any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011147628.6A CN112351079B (en) | 2020-10-23 | 2020-10-23 | Space application and data integrated packaging system and method based on data box |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011147628.6A CN112351079B (en) | 2020-10-23 | 2020-10-23 | Space application and data integrated packaging system and method based on data box |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112351079A CN112351079A (en) | 2021-02-09 |
| CN112351079B true CN112351079B (en) | 2021-10-22 |
Family
ID=74360030
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011147628.6A Active CN112351079B (en) | 2020-10-23 | 2020-10-23 | Space application and data integrated packaging system and method based on data box |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112351079B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113568658B (en) * | 2021-08-13 | 2023-09-19 | 中国科学院西北生态环境资源研究院 | Multi-language geoscience online service method and device, storage medium and electronic equipment |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103780682A (en) * | 2014-01-07 | 2014-05-07 | 中国船舶重工集团公司第七0九研究所 | Information sharing system for Ad Hoc network |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| AU2011221678B2 (en) * | 2010-03-05 | 2014-08-14 | Samsung Electronics Co., Ltd. | Method and apparatus for generating and reproducing adaptive stream based on file format, and recording medium thereof |
| KR101781873B1 (en) * | 2010-04-19 | 2017-09-26 | 엘지전자 주식회사 | Apparatus and method for transmitting and receiving contents based on internet |
| US9264720B2 (en) * | 2010-10-08 | 2016-02-16 | Samsung Electronics Co., Ltd | Apparatus and method for multimedia service |
| CN107633181B (en) * | 2017-09-12 | 2021-01-26 | 复旦大学 | Data model realization method facing data open sharing and operation system thereof |
-
2020
- 2020-10-23 CN CN202011147628.6A patent/CN112351079B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103780682A (en) * | 2014-01-07 | 2014-05-07 | 中国船舶重工集团公司第七0九研究所 | Information sharing system for Ad Hoc network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112351079A (en) | 2021-02-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109492040B (en) | System suitable for processing mass short message data in data center | |
| US10664331B2 (en) | Generating an application programming interface | |
| CN111984830A (en) | Management operation and maintenance platform and data processing method | |
| US11016954B1 (en) | Distributed data set extraction for migration | |
| CN101621405B (en) | Distributed type management monitoring system, monitoring method and establishing method thereof | |
| US10558649B2 (en) | Database server system monitoring | |
| CN110431545A (en) | Inquiry is executed for structural data and unstructured data | |
| CN111143054B (en) | Heterogeneous domestic CPU resource fusion management method | |
| CN103399942B (en) | A kind of data engine system supporting the many tenants of SaaS and method of work thereof | |
| CN111563018B (en) | Resource management and monitoring method of man-machine-object fusion cloud computing platform | |
| CN106548288B (en) | The polymorphic instance management system and method for the more scenes of electric power | |
| CN111641676B (en) | Method and device for constructing third-party cloud monitoring service | |
| CN103957270A (en) | Cloud atomic unit delivery and deployment method and device | |
| CN111552010A (en) | Satellite remote sensing quick response and visualization service platform for global meteorological disasters | |
| CN116680040A (en) | Container processing method, device, equipment, storage medium and program product | |
| CN112351079B (en) | Space application and data integrated packaging system and method based on data box | |
| CN113127526A (en) | Distributed data storage and retrieval system based on Kubernetes | |
| CN104301354B (en) | A kind of method and system of the spatial class business datum GISization service based on cloud computing | |
| CN114969199A (en) | Method, device and system for processing remote sensing data and storage medium | |
| CN113468275A (en) | Data importing method and device of graph database, storage medium and electronic equipment | |
| CN115878121A (en) | Terminal code increment compiling method, system, device, server and storage medium | |
| CN102761570A (en) | System and method for monitoring grid resources based on agents | |
| CN111949378A (en) | Virtual machine starting mode switching method and device, storage medium and electronic equipment | |
| Lynnes et al. | Archive management of NASA earth observation data to support cloud analysis | |
| US20230153302A1 (en) | Optimizing query performance in virtual database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |