CN114253685A - Calling framework and mode of remote sensing image algorithm - Google Patents
Calling framework and mode of remote sensing image algorithm Download PDFInfo
- Publication number
- CN114253685A CN114253685A CN202111511840.0A CN202111511840A CN114253685A CN 114253685 A CN114253685 A CN 114253685A CN 202111511840 A CN202111511840 A CN 202111511840A CN 114253685 A CN114253685 A CN 114253685A
- Authority
- CN
- China
- Prior art keywords
- algorithm
- cluster
- remote sensing
- sensing image
- calling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/52—Program synchronisation; Mutual exclusion, e.g. by means of semaphores
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a remote sensing image algorithm calling architecture and a remote sensing image algorithm calling mode, which comprise a Docker container cluster, a database cluster, a Dubbo service cluster and a Nginx server; the system comprises a Docker container, a database cluster, a Dubbo service cluster and an Nginx server, wherein the Docker container contains a remote sensing image algorithm and a unified algorithm management micro-service, the database cluster is used for storing an algorithm calling state and an algorithm execution result, and the Nginx server is used for receiving an algorithm service request of the Dubbo service cluster; the Dubbo micro-service cluster sends an algorithm service calling request to the Nginx server, calls an algorithm from the Docker container, and is also used for acquiring an algorithm result path and an algorithm calling state from the database cluster. The Docker container corresponding to the algorithm is added, the load balancing capacity of the Nginx is utilized, the problem of high concurrency caused by increase of the access amount is solved, and compared with a single framework, the maintenance and expansion capacity is improved.
Description
Technical Field
The invention belongs to the field of data processing, and particularly relates to a calling framework and a calling mode of a remote sensing image algorithm based on Docker and Nginx.
Background
The remote sensing technology is a technology for detecting and identifying a target by sensing electromagnetic waves, visible light and infrared rays reflected by the target or radiated by the target from a long distance. With the continuous development of remote sensing technology, the remote sensing technology has wide application in various fields such as agriculture, geology, oceans, meteorological mapping, environmental protection, disaster prevention and relief, military and the like. The image algorithm plays an increasingly important role as an indelible link of remote sensing image application.
In the traditional single system, the remote sensing image algorithm is deployed on the corresponding virtual machine, and the running environment, the dependency library and the like of the corresponding algorithm are added. As more and more users come, the concurrency probability borne by programs is improved, but the concurrency capability of the single system is limited, so that the traditional single system structure cannot meet the use of the remote sensing image algorithm in various situations. In addition, as the user requirements become more complex, besides increasing the code amount of the system, the readability, maintainability and expandability of the code also decrease, and the modification service of the monolithic system may bring influence to other services, resulting in increased testing difficulty.
Disclosure of Invention
In view of the above-mentioned drawbacks and deficiencies of the prior art, the present invention is directed to a remote sensing image algorithm calling architecture and method.
In order to achieve the purpose, the implementation of the invention adopts the following technical scheme:
the first aspect provides a calling architecture of a remote sensing image algorithm, which comprises a Docker container cluster, a database cluster, a Dubbo service cluster and a Nginx server, wherein the Docker container cluster is in communication connection with the database cluster; each Docker container in the Docker container cluster contains a plurality of remote sensing image algorithms with the same environment or the same service requirement and unified algorithm management micro-services, the database cluster consists of a plurality of databases used for storing algorithm calling states and algorithm result paths, the Dubbo service cluster consists of Dubbo services, the Dubbo services are used for reading calling state information and result storage paths, and the Nginx server is used for receiving algorithm service requests of the Dubbo service cluster; and the Dubbo service cluster is used for acquiring an algorithm result path and an algorithm calling state from the database cluster after the algorithm calling is successful.
In a second aspect, a remote sensing image algorithm calling method is provided, which adopts the calling architecture described above, and includes the following steps: s1, loading a plurality of remote sensing image algorithms with the same environment or the same service requirement into a plurality of corresponding Docker containers, and configuring a unified algorithm management microservice; step S2, based on the load balancing algorithm, selecting an algorithm in a Docker container from the address list of the service provider for the user to call; step S3, monitoring the algorithm execution result, and storing the execution state and the result path into a database of the database cluster; step S4, the Dubbo service cluster obtains the execution success status and the algorithm execution result under the corresponding path.
According to the technical scheme provided by the embodiment of the application, the step S1 includes the following steps: step S11, loading a plurality of remote sensing image algorithms and unified algorithm management micro-services into a plurality of corresponding Docker containers as individual services; step S12, packaging Docker containers with a plurality of remote sensing image algorithm services into mirror images, and deploying the mirror images into other Docker containers; and step S13, configuring the address of the remote sensing image algorithm calling service in Nginx.
According to the technical scheme provided by the embodiment of the application, the unified configuration of the remote sensing image algorithm loaded into the Docker container comprises the following steps: input and output of an algorithm are specified, unified algorithm management microservices are configured, and a database table is configured to store algorithm execution states and algorithm execution results; receiving an algorithm execution request, and storing an algorithm execution state and a result storage path in a database; and when the algorithm is called, updating and writing the state file into the database table, and after the calling is finished, updating the operation record and the order state of the database table for displaying.
According to the technical scheme provided by the embodiment of the application, the synchronous lock processing is executed while the user calls the algorithm based on load balance selection until the call is finished; and the server reads the algorithm execution state information, judges the state, releases the synchronous lock, and updates the order flow or starts to process the next request.
According to the technical scheme provided by the embodiment of the application, if one Docker container is failed to call or is down, the server is automatically switched to the server of other service providers to retry for the preset times according to the failure retry set by the cluster fault-tolerant strategy.
The invention has the following beneficial effects:
according to different application scenes, the remote sensing image algorithm has obvious difference on the data volume required by calculation, the corresponding execution time is increased when the data volume is large, the execution time of the algorithm is influenced, the operations of splicing, blending and cutting the image are also included, or data conversion such as the original data h5 file is converted into a tiff file through the algorithm, the tasks can be grouped at a calling end, and then the same algorithm in a plurality of docker containers is called to save time. If the calling is a single thread, when a plurality of algorithm calling services are executed, the next procedure can be executed only after the last procedure is finished, the use effect is not very friendly, the more complex the service is, the readability, maintainability and expandability of the code are reduced, and the modification of the service by the single application may affect other services, so that the test difficulty is increased; based on the situation of the prior art, the application provides a structure adopting Docker and Nginx, which can increase Docker containers corresponding to algorithms, and simultaneously utilizes load balancing capacity of Nginx to deal with high concurrency capacity brought by access increase. Meanwhile, the problems of time loss, incompatibility and the like caused by the fact that the algorithm is configured in a specific operation environment at a server side are solved, and the specific algorithm can be expanded according to requirements. If one container is failed to call or is down, setting failure retry according to the cluster fault-tolerant strategy, and automatically switching to servers of other service providers to retry.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments made with reference to the following drawings:
FIG. 1 is a block diagram of a remote sensing image algorithm invocation architecture according to the present application;
fig. 2 and fig. 3 are flowcharts of remote sensing image algorithm calling methods according to the present application.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the present invention are shown in the drawings.
It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict. The present invention will be described in detail below with reference to the embodiments with reference to the attached drawings.
A calling architecture of a remote sensing image algorithm comprises a Docker container cluster, a database cluster, a Dubbo service cluster and a Nginx server, wherein the Docker container cluster is in communication connection with the database cluster; the system comprises a Docker container cluster formed by a plurality of Docker containers, a database cluster and a server, wherein each Docker container in the Docker container cluster contains a plurality of remote sensing image algorithms with the same environment or the same service requirement and unified algorithm management micro-services, the database cluster consists of a plurality of databases used for storing algorithm calling states and algorithm execution results, the Dubbo service cluster consists of Dubbo services, the Dubbo services are used for reading calling state information and result storage paths, and an Nginx server is used for receiving algorithm service requests of the Dubbo service cluster; the Dubbo micro-service cluster is used for calling algorithm service when requesting to the Nginx server, calling the algorithm from the Docker container, and obtaining an algorithm result path and an algorithm calling state from the database cluster. The Docker container is also used for the Dubbo service to read the storage path and obtain the algorithm execution result.
Specifically, the application provides a framework of a remote sensing image algorithm calling mode based on Docker and Nginx, as shown in fig. 1, the framework of the remote sensing image algorithm calling mode based on Docker and Nginx includes a Docker container cluster formed by a plurality of Docker containers, a database cluster formed by a plurality of databases for storing calling states, a Dubbo service cluster formed by Dubbo for reading calling state information, and a Nginx server for receiving an algorithm calling request initiated by the Dubbo service cluster. The remote sensing image algorithms have the same dependence environment and are configured with a unified algorithm management microservice. After receiving the algorithm calling request sent by the Dubbo, Nginx calls the algorithm from the Docker container and returns an algorithm calling address. The Dubbo service responds to the algorithm execution results from the Docker container group. Therefore, Docker containers corresponding to the algorithms can be increased, high concurrency caused by increase of access amount is responded by using load balancing capacity of Nginx, and maintenance and expansion capacity is improved compared with a single framework.
Specifically, a micro-service in the Dubbo service cluster requests Nginx to call an algorithm; the Dubbo service calling the algorithm selects a Docker container and calls the algorithm; the Docker container stores the state information into a database cluster formed by a database and stores the operation result of the algorithm into a specific path such as a shared folder; and the Dubbo service cluster formed by the Dubbo service end reads the state information from the database and acquires the calling result from the Docker container.
A calling mode of a remote sensing image algorithm adopts the calling framework, and comprises the following steps: step S1, loading a plurality of remote sensing image algorithms with the same environment or the same service requirement into a plurality of corresponding Docker containers, and configuring a unified algorithm management microservice; step S2, based on the load balancing algorithm, selecting an algorithm in a Docker container from the address list of the service provider for the user to call; step S3, monitoring the algorithm executing structure, and storing the executing state and the result path into the database of the database cluster; step S4, the Dubbo service cluster obtains the execution success status and the algorithm execution result under the corresponding path.
Specifically, as shown in fig. 2, a plurality of remote sensing image algorithms are loaded into a plurality of corresponding Docker containers. In the embodiment of the application, the remote sensing image algorithm refers to an atmospheric aerosol and atmospheric water content related algorithm, such as an atmospheric main greenhouse gas monitor data simulation algorithm, an atmospheric aerosol pixel conversion algorithm and the like. Therefore, the model operation has asynchronous or parallel computation capability due to the multiple Docker containers with multiple algorithms, the efficiency of the model batch operation is improved, and the system is more stable and robust. The unified configuration of the algorithm means that the input and output of the algorithm are standardized, and the server side can conveniently read the structure to perform the next operation. After the calling state is stored, the operation record and the state can be conveniently updated subsequently, and the calling times can be counted outside the operation process.
Further, the step S1 of loading the plurality of remote sensing image algorithms into the plurality of corresponding Docker containers includes the following steps: s11, loading a plurality of remote sensing image algorithms and unified algorithm management micro-services into a plurality of corresponding Docker containers as individual services; step S12, packaging Docker containers with a plurality of remote sensing image algorithm services into mirror images, and deploying the mirror images into other Docker containers; and step S13, configuring the address of the remote sensing image algorithm strip service in Nginx.
Specifically, a mirror image deployment mode is adopted, so that the condition that the settings of each Docker are the same is ensured, the deployment is rapid and accurate, the algorithm deployment time is saved, the configuration of the algorithm depending on a complex environment is facilitated, and the expansion capability of the system is further improved.
Further, the algorithm management microservice for uniformly configuring the remote sensing image algorithm loaded into the Docker container comprises the following steps: input and output of an algorithm are specified, and a database table storage state file and an algorithm result path are configured; receiving an algorithm execution request, and storing an algorithm execution state and a result storage path in a database; and when the algorithm is called, updating and writing the state file into the database table, and after the calling is finished, updating the operation record and the order state of the database table for displaying.
Specifically, the algorithm management microservices are uniformly configured, a database table is configured, the state and algorithm result storage paths are written into the database table, the database table is read by the server side at regular time for judgment, and algorithm entering and exiting are normalized, so that the monitoring of the algorithm execution state or the judgment of whether the algorithm execution state is finished or not is facilitated, and the server side can read the result for the next operation.
Further, the algorithm is selected based on the load balancing algorithm for the user to call, and meanwhile, synchronous lock processing is executed until the call is finished; and the server reads the file of the algorithm execution state, judges the state, releases the synchronous lock, and updates the order flow or starts to process the next request.
Specifically, the algorithm is selected for the user to call, and meanwhile, synchronous lock processing is executed until the call is finished; the server reads the state file executed by the algorithm, and then the synchronization lock can be released, and the order flow is updated or the next request is processed.
Further, if one Docker container fails to be called or is down, failure retry is set according to the cluster fault-tolerant strategy, and servers of other service providers are automatically switched to for retry for a preset number of times.
Specifically, due to the fact that a plurality of Docker containers are processed in parallel, one container is failed to be called or is down, and the other containers are automatically switched to be called, so that the fault tolerance of the calling mode is enhanced, and the calling efficiency is improved.
The foregoing description is only exemplary of the preferred embodiments of the invention and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention herein disclosed is not limited to the particular combination of features described above, but also encompasses other arrangements formed by any combination of the above features or their equivalents without departing from the spirit of the invention. For example, the above features and (but not limited to) features having similar functions disclosed in the present invention are mutually replaced to form the technical solution.
Claims (6)
1. A calling architecture of a remote sensing image algorithm is characterized by comprising a Docker container cluster, a database cluster, a Dubbo service cluster and a Nginx server, wherein the Docker container cluster is in communication connection with the database cluster; each Docker container in the Docker container cluster contains a plurality of remote sensing image algorithms with the same environment or the same service requirement and unified algorithm management micro-services, the database cluster consists of a plurality of databases used for storing algorithm calling states and algorithm result paths, the Dubbo service cluster consists of Dubbo services, the Dubbo services are used for reading calling state information and result storage paths, and the Nginx server is used for receiving algorithm service requests of the Dubbo service cluster; the Dubbo service cluster is also used for acquiring an algorithm result path and an algorithm calling state from the database cluster after the algorithm calling is successful.
2. A remote sensing image algorithm calling method is characterized in that the calling framework of claim 1 is adopted, and the method comprises the following steps:
s1, loading a plurality of remote sensing image algorithms with the same environment or the same service requirement into a plurality of corresponding Docker containers, and configuring a unified algorithm management microservice;
step S2, based on the load balancing algorithm, selecting an algorithm in a Docker container from the address list of the service provider for the user to call;
step S3, monitoring the algorithm execution result, and storing the execution state and the result path into a database of the database cluster;
step S4, the Dubbo service cluster obtains the execution success status and the algorithm execution result under the corresponding path.
3. The method for invoking the remote sensing image algorithm according to claim 2, wherein step S1 comprises the steps of:
step S11, loading a plurality of remote sensing image algorithms and unified algorithm management micro-services into a plurality of corresponding Docker containers as individual services;
step S12, packaging Docker containers with a plurality of remote sensing image algorithm services into mirror images, and deploying the mirror images into other Docker containers;
and step S13, configuring the address of the remote sensing image algorithm calling service in Nginx.
4. The method for invoking the remote sensing image algorithm according to claim 2, wherein the step of configuring the uniform algorithm management microservice for the remote sensing image algorithm loaded into the Docker container comprises the steps of:
input and output of an algorithm are specified, and a database table storage state file and an algorithm result path are configured;
receiving an algorithm execution request, and storing an algorithm execution state and a result storage path in a database;
and when the algorithm is called, updating and writing the state file into the database table, and after the calling is finished, updating the operation record and the order state of the database table for displaying.
5. The method of claim 4, wherein the remote sensing image algorithm is invoked,
executing synchronous lock processing while providing for the user to call based on the load balancing selection algorithm until the call is finished;
and the server reads the information of the algorithm execution state, judges the state, releases the synchronous lock, and updates the order flow or starts to process the next request.
6. A method as claimed in claim 2, wherein if a Docker container fails to be called or goes down, a retry of the call is set according to the cluster fault tolerance policy, and the server of another service provider is automatically switched to perform a predetermined number of retries.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111511840.0A CN114253685A (en) | 2021-12-06 | 2021-12-06 | Calling framework and mode of remote sensing image algorithm |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111511840.0A CN114253685A (en) | 2021-12-06 | 2021-12-06 | Calling framework and mode of remote sensing image algorithm |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114253685A true CN114253685A (en) | 2022-03-29 |
Family
ID=80794843
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111511840.0A Pending CN114253685A (en) | 2021-12-06 | 2021-12-06 | Calling framework and mode of remote sensing image algorithm |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114253685A (en) |
-
2021
- 2021-12-06 CN CN202111511840.0A patent/CN114253685A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20190340168A1 (en) | Merging conflict resolution for multi-master distributed databases | |
| US7555541B2 (en) | Method and apparatus for managing configuration information in a distributed computer system | |
| CN102088490B (en) | Data storage method, device and system | |
| US8402119B2 (en) | Real-load tuning of database applications | |
| CN114301972B (en) | Hierarchical deployment method and system for blockchain nodes based on cloud edge cooperation | |
| CN113360456B (en) | Data archiving method, device, equipment and storage medium | |
| CN106569896B (en) | A kind of data distribution and method for parallel processing and system | |
| CN112559637B (en) | Data processing method, device, equipment and medium based on distributed storage | |
| CN109819004A (en) | For disposing the method and system at more live data centers | |
| US20210326359A1 (en) | Compare processing using replication log-injected compare records in a replication environment | |
| CN111478828A (en) | Pressure testing method, device and system for block chain network | |
| CN113934539B (en) | Construction method of geographic information service system based on micro-service architecture | |
| JP6975153B2 (en) | Data storage service processing method and equipment | |
| CN111258742B (en) | Data synchronization method, system, computing device and storage medium | |
| CN111143331A (en) | Data migration method and device and computer storage medium | |
| CN109614270A (en) | Data read-write method, device, equipment and storage medium based on Hbase | |
| CN113326161A (en) | Root cause analysis method | |
| CN114745295A (en) | Data acquisition method, device, equipment and readable storage medium | |
| CN112788124B (en) | Remote sensing image distributed registration service method and device | |
| CN108664343A (en) | A kind of stateful call method and device of micro services | |
| WO2020259191A1 (en) | Data centre node allocation method, apparatus, and system and computer device | |
| CN114253685A (en) | Calling framework and mode of remote sensing image algorithm | |
| CN116383223A (en) | Asset data processing method, related device and storage medium | |
| CN115587141A (en) | Database synchronization method and device | |
| CN112527463B (en) | Container mirror image downloading acceleration method based on object storage |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication |