CN112764894A - Credit generation analysis task scheduling system based on container technology, and construction method and scheduling scheme thereof - Google Patents
Credit generation analysis task scheduling system based on container technology, and construction method and scheduling scheme thereof Download PDFInfo
- Publication number
- CN112764894A CN112764894A CN202011465097.5A CN202011465097A CN112764894A CN 112764894 A CN112764894 A CN 112764894A CN 202011465097 A CN202011465097 A CN 202011465097A CN 112764894 A CN112764894 A CN 112764894A
- Authority
- CN
- China
- Prior art keywords
- task
- analysis
- mirror image
- module
- task scheduling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000004458 analytical method Methods 0.000 title claims abstract description 147
- 238000005516 engineering process Methods 0.000 title claims abstract description 18
- 238000010276 construction Methods 0.000 title abstract description 8
- 239000002699 waste material Substances 0.000 claims abstract description 6
- 238000000034 method Methods 0.000 claims description 25
- 240000007087 Apium graveolens Species 0.000 claims description 13
- 235000015849 Apium graveolens Dulce Group Nutrition 0.000 claims description 13
- 235000010591 Appio Nutrition 0.000 claims description 13
- 230000000007 visual effect Effects 0.000 claims description 7
- 238000012217 deletion Methods 0.000 claims description 6
- 230000037430 deletion Effects 0.000 claims description 6
- 238000007726 management method Methods 0.000 claims description 6
- 238000012544 monitoring process Methods 0.000 claims description 6
- 238000002955 isolation Methods 0.000 claims description 4
- 238000012550 audit Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 claims description 3
- 238000012384 transportation and delivery Methods 0.000 claims description 3
- 230000010076 replication Effects 0.000 claims 1
- 239000000126 substance Substances 0.000 claims 1
- 238000013468 resource allocation Methods 0.000 abstract description 3
- 230000001737 promoting effect Effects 0.000 abstract description 2
- 238000009434 installation Methods 0.000 description 8
- 238000012545 processing Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 2
- 238000001125 extrusion Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
- G06F8/61—Installation
- G06F8/63—Image based installation; Cloning; Build to order
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5077—Logical partitioning of resources; Management or configuration of virtualized resources
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45562—Creating, deleting, cloning virtual machine instances
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/4557—Distribution of virtual machine instances; Migration and load balancing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Stored Programmes (AREA)
Abstract
The invention provides a letter generation analysis task scheduling system based on a container technology. The invention not only solves the problem of difficult construction of the analysis environment, but also has the advantages of portability, one-time construction and multiple-use. The credit generation analysis task scheduling system is used for effectively managing the credit generation analysis tasks, is convenient for iterative upgrade of the analysis tasks, supports real-time checking of the running states of the tasks, greatly avoids the waste and the squeezing condition of computing resources, greatly reduces the working hours of the credit generation analysis personnel for repeatedly building credit generation analysis environments, effectively ensures the stability of the analysis tasks and reduces the cost of enterprise operation. The invention also provides a letter generation analysis task scheduling scheme based on the system, and the task scheduling scheme is used for greatly optimizing resource allocation and promoting the improvement of productivity.
Description
Technical Field
The invention belongs to the technical field of biological information data processing, and relates to a letter generation analysis task scheduling system based on a container technology and a construction method thereof.
Background
The container technology is to divide the resources of a single operating system into isolated groups and solve the problem of resource use conflict between the isolated groups. Docker is a lightweight and open-source container engine widely applied at present, runs in Linux and Windows systems, and creates a lightweight and portable container meeting the application running requirements for any application under the condition of almost no performance overhead.
Celery is a distributed task queue that focuses on real-time processing and task scheduling, and is simple, flexible, and reliable. When an operation triggered by the user takes a long time to execute, the Celery asynchronously executes the operation as a task, and returns the result to the user after the execution is finished.
In the daily biological information data analysis process, the dependence of software on the system and the dependence of analysis processes on the software are different, each analysis process can only run on a specific host, the waste of computing resources can be caused when the task of the same analysis process is less, and the squeezing phenomenon of the computing resources can be caused when the task is more. The Docker mirror image technology effectively solves the problem of different analysis environments, analysts can build personalized version mirror images on a host computer provided with a Docker engine by compiling Docker file files of the analysis environments, and start containers based on the mirror images to perform analysis tasks. Docker uses a container to run an analysis process, is not limited by an operating system, and avoids the problem of conflict between system dependence and software dependence. Celery's distributed task queue technique has well solved the problem of system resource crowded when the task is more, and Celery both can avoid system resource crowded and convert through the concurrent quantity of setting up the task operation, the mode that the surplus task was queued, also can solve this problem through distributing some analysis tasks to other host computer analysis, very big promotion analysis environment's stability.
Disclosure of Invention
In order to solve the defects of the prior art, the invention aims to provide a credit generation analysis task scheduling system based on a container technology, which solves the problem of building different credit generation analysis task environments, reasonably distributes resource operation tasks through the task scheduling system, and avoids the extrusion and waste of system resources.
The invention relates to a letter generation analysis task scheduling system based on a container technology, which comprises an analysis mirror image warehouse module, a mirror image intelligent pulling module, a task scheduling module and a task running module.
The analysis mirror image warehouse module is an enterprise-level Registry-based open-source project, is divided into a user mirror image warehouse and a system mirror image warehouse, is used for storing a student-trust analysis mirror image constructed by an analyst, can realize the isolation of projects and user authorities, comprises master-slave copy logic for customizing user warehouses and system warehouse backups, can also isolate sensitive operations of the user warehouses and the system warehouses, and provides high-level security characteristics such as user management, access control, activity audit and the like. In order to meet the requirements of the letter generation analysis task scheduling system, personalized customization is performed, and the personalized customization is used for storing Docker image files of various analysis processes and various versions.
The mirror image intelligent pulling module is used for receiving a task distributed by the task scheduling module at a local host, automatically searching whether the current local host contains a Docker mirror image required by the analysis task, searching an analysis mirror image system warehouse if the current local host does not have the Docker mirror image specified by the analysis task, automatically pulling the mirror image to the current host when the specified Docker mirror image exists in the system mirror image warehouse, executing the task operation module, and stopping task operation and feeding back if the specified Docker mirror image does not exist.
The task scheduling module is customized and built based on an open source project Celery, and the Celery is a simple, flexible and reliable distributed queue system for processing a large number of messages, is dedicated to processing asynchronous task queues in real time and supports task scheduling. The battery mainly comprises a message middleware (message browser), a task execution unit (worker) and a task result storage unit (task result store). After the credit generation analysis task is submitted, the task is distributed and delivered through the task scheduling module.
And the task running module starts to execute the credit generation analysis task under the condition that the task distributed by the task scheduling module is received and the Docker mirror image specified by the analysis task exists in the current host, automatically stores the analysis result after the task runs successfully, and prints error report information of the running failure of the analysis task if the task fails.
The invention also provides a method for constructing the letter generation analysis task scheduling system based on the container technology, which mainly comprises the following implementation steps:
the method comprises the following steps: installing a Docker engine on a host computer needing to run an analysis task, and starting the Docker engine;
step two: building an analysis mirror image warehouse module, wherein the analysis mirror image warehouse module is built based on a Harbor, and the Harbor is an open source solution for building an enterprise-level private Docker mirror image warehouse;
downloading an offline installation package by the Harbor official network, decompressing the installation package and modifying the Harbor installation configuration; after the installation configuration is modified, a docker-composition is used for starting a Harbor service, configuring user rights and project rights of a Harbor warehouse, configuring a service for master-slave copy (a user mirror image warehouse is automatically synchronized to a system mirror image warehouse at regular intervals), and analyzing the backup of a mirror image;
step three: compiling a Dockerfile file of a system environment and software which depend on a letter analysis process, constructing an individualized version mirror image, and uploading the individualized version mirror image to a user mirror image warehouse, wherein the content of the user mirror image warehouse can be automatically updated to the system mirror image warehouse, but the deletion operation of the user mirror image warehouse is asynchronous with the system mirror image warehouse;
only an administrator has the authority of deleting operation in the system mirror image warehouse, so that the stability of an analysis environment is ensured, and the execution failure of a trust analysis task caused by mistaken deletion of an analysis mirror image is prevented;
step four: the method comprises the following steps of constructing a task scheduling module which mainly comprises three parts, namely a message middleware, a task execution unit and a task result storage unit;
the message middleware part integrates Redis as message middleware for delivering and distributing task messages because the Celery does not provide the message middleware;
the task execution unit is responsible for executing specific tasks and can run in a distributed system node concurrently;
the task result storage unit is mainly responsible for storing the operation result of the task execution unit;
the task scheduling module can carry out health check on the resource condition of each host system in the current cluster environment and select a host with proper resource for task delivery; the task scheduling module is also integrated with a monitoring tool Flower of the Celery, monitors the execution condition and the health condition of each credit generation analysis task and presents the execution condition and the health condition in a visual mode;
step five: and constructing a task operation module, compiling a visual webpage front-end interface and supporting the filling of analysis information.
The analyst writes the analysis content and submits a task, and the task running module automatically analyzes the analysis content into a Docker execution command;
the task scheduling system obtains a Docker execution command of an analysis task and automatically transmits the Docker execution command to the task scheduling module, and the task scheduling module intelligently selects a host with proper resources to perform credit generation analysis tasks according to the resource condition of the cluster system;
after the letter generation analysis task is executed and a generated result is obtained, the task scheduling system stores the operation result of the execution unit, the task operation module is responsible for collecting and storing a letter generation analysis result file, and letter generation analysis personnel can directly download and check the letter generation analysis result.
The invention also provides a letter generation analysis task scheduling scheme based on the system.
The invention has the beneficial effects that: the problem that the analysis environment is difficult to build is solved based on the container technology, the method has the advantages of portability, one-time building and multiple-place use, and the updating and iteration efficiency of the raw letter analysis task is greatly improved. And the task scheduling system is used for carrying out task management on the credit analysis tasks, so that the waste or the extrusion of resources is greatly avoided, and the stability of the analysis tasks is effectively ensured. The scheduling scheme of the raw trust analysis task greatly optimizes resource allocation and promotes the improvement of productivity.
Drawings
FIG. 1 is a diagram of a system architecture for scheduling a credit generation analysis task based on container technology;
FIG. 2 is a flow chart for constructing a letter generation analysis task scheduling system based on a container technology;
FIG. 3 is a set up analysis mirror repository module UI interface;
FIG. 4 is a host Docker mirror management background;
FIG. 5 is a diagram showing the results of the task execution module.
Detailed Description
The present invention will be described in further detail with reference to the following specific examples and the accompanying drawings. The procedures, conditions, experimental methods and the like for carrying out the present invention are general knowledge and common general knowledge in the art except for the contents specifically mentioned below, and the present invention is not particularly limited.
The invention provides a letter generation analysis task scheduling system based on a container technology and a construction method thereof. The invention not only solves the problem of difficult construction of the analysis environment, but also has the advantages of portability, one-time construction and multiple-use. The credit generation analysis task scheduling system is used for effectively managing the credit generation analysis tasks, is convenient for iterative upgrade of the analysis tasks, supports real-time checking of the running states of the tasks, greatly avoids the waste and the squeezing condition of computing resources, greatly reduces the working hours of the credit generation analysis personnel for repeatedly building credit generation analysis environments, effectively ensures the stability of the analysis tasks and reduces the cost of enterprise operation. The invention also provides a letter generation analysis task scheduling scheme based on the system, and the task scheduling scheme is used for greatly optimizing resource allocation and promoting the improvement of productivity.
The invention provides a letter generation analysis task scheduling system based on a container technology, which comprises an analysis mirror image warehouse module, a mirror image intelligent pulling module, a task scheduling module and a task running module.
The analysis mirror image warehouse module is an enterprise-level Registry-based open-source project, is divided into a user mirror image warehouse and a system mirror image warehouse, is used for storing a student-trust analysis mirror image constructed by an analyst, can realize the isolation of projects and user authorities, comprises master-slave copy logic for customizing user warehouses and system warehouse backups, can also isolate sensitive operations of the user warehouses and the system warehouses, and provides high-level security characteristics such as user management, access control, activity audit and the like. In order to meet the requirements of the letter generation analysis task scheduling system, personalized customization is performed, and the personalized customization is used for storing Docker image files of various analysis processes and various versions.
The mirror image intelligent pulling module is used for receiving a task distributed by the task scheduling module at a local host, automatically searching whether the current local host contains a Docker mirror image required by the analysis task, searching an analysis mirror image system warehouse if the current local host does not have the Docker mirror image specified by the analysis task, automatically pulling the mirror image to the current host when the specified Docker mirror image exists in the system mirror image warehouse, executing the task operation module, and stopping task operation and feeding back if the specified Docker mirror image does not exist.
The task scheduling module is customized and built based on an open source project Celery, and the Celery is a simple, flexible and reliable distributed queue system for processing a large number of messages, is dedicated to processing asynchronous task queues in real time and supports task scheduling. The battery mainly comprises a message middleware (message browser), a task execution unit (worker) and a task result storage unit (task result store). After the credit generation analysis task is submitted, the task is distributed and delivered through the task scheduling module.
And the task running module starts to execute the credit generation analysis task under the condition that the task distributed by the task scheduling module is received and the Docker mirror image specified by the analysis task exists in the current host, automatically stores the analysis result after the task runs successfully, and prints error report information of the running failure of the analysis task if the task fails.
The invention also provides a method for constructing the letter generation analysis task scheduling system based on the container technology, which mainly comprises the following implementation steps:
the method comprises the following steps: installing a Docker engine on a host computer needing to run an analysis task, and starting the Docker engine;
step two: building an analysis mirror image warehouse module, wherein the analysis mirror image warehouse module is built based on a Harbor, and the Harbor is an open source solution for building an enterprise-level private Docker mirror image warehouse;
downloading an offline installation package by the Harbor official network, decompressing the installation package and modifying the Harbor installation configuration, wherein the configuration to be modified mainly comprises a host name hostname, https configuration, a password of an admin account, database configuration, a data persistence directory and the like;
after the installation configuration is modified, a docker-composition is used for starting a Harbor service, configuring user rights and project rights of a Harbor warehouse, configuring a service for master-slave copy (a user mirror image warehouse is automatically synchronized to a system mirror image warehouse at regular intervals), and analyzing the backup of a mirror image;
step three: compiling a Dockerfile file of a system environment and software which depend on a letter analysis process, constructing an individualized version mirror image, and uploading the individualized version mirror image to a user mirror image warehouse, wherein the content of the user mirror image warehouse can be automatically updated to the system mirror image warehouse, but the deletion operation of the user mirror image warehouse is asynchronous with the system mirror image warehouse;
only an administrator has the authority of deleting operation in the system mirror image warehouse, so that the stability of an analysis environment is ensured, and the execution failure of a trust analysis task caused by mistaken deletion of an analysis mirror image is prevented;
step four: a task scheduling module is built, and the task module mainly comprises three parts, including a message middleware, a task execution unit and a task result storage unit;
the message middleware part integrates Redis as message middleware for delivering and distributing task messages because the Celery does not provide the message middleware;
the task execution unit is responsible for executing specific tasks and can run in a distributed system node concurrently;
the task result storage unit is mainly responsible for storing the operation result of the task execution unit;
the task scheduling system can carry out health check on the resource condition of each host system in the current cluster environment and select a host with proper resource for task delivery;
the task scheduling system also integrates a monitoring tool Flower of the Celery, monitors the conditions and health conditions of various credit generation analysis tasks and presents the conditions and health conditions in a visual mode;
step five: and constructing a task operation module, compiling a visual webpage front-end interface and supporting the filling of analysis information.
The analyst writes the analysis content and submits a task, and the task running module automatically analyzes the analysis content into a Docker execution command;
the task scheduling system obtains a Docker execution command of an analysis task and automatically transmits the Docker execution command to the task scheduling module, and the task scheduling module intelligently selects a host with proper resources to perform a credit generation analysis task according to the resource condition of the cluster system;
after the letter generation analysis task is executed and a generated result is obtained, the task scheduling system stores the operation result of the execution unit, the task operation module is responsible for collecting and storing a letter generation analysis result file, and letter generation analysis personnel can directly download and check the letter generation analysis result.
The invention also provides a letter generation analysis task scheduling scheme based on the system.
The protection of the present invention is not limited to the above embodiments. Variations and advantages that may occur to those skilled in the art may be incorporated into the invention without departing from the spirit and scope of the inventive concept, which is set forth in the following claims.
Claims (10)
1. A system for scheduling a credit generation analysis task based on container technology, comprising: the system comprises an analysis mirror image warehouse module, a mirror image intelligent pulling module, a task scheduling module and a task running module; wherein the content of the first and second substances,
the analysis mirror image warehouse module is divided into a user mirror image warehouse and a system mirror image warehouse and is used for storing a student information analysis mirror image constructed by an analyst, supporting project and user authority isolation, providing advanced safety characteristics of user management, access control and activity audit, and carrying out master-slave backup on the system mirror image warehouse and the user mirror image warehouse at regular time;
the intelligent mirror image pulling module is used for automatically detecting whether the current local host has the specified analysis mirror image required by the credit generation analysis, automatically searching the system mirror image warehouse if the local host does not have the analysis mirror image, automatically pulling the local host if the specified analysis mirror image is detected to exist in the system mirror image warehouse, and directly stopping the running of the credit generation analysis task if the specified analysis mirror image does not exist;
the task scheduling module calls an internal script to read the environment variable configuration of the Celery of the distributed system, calls a preset task scheduling template, generates a new task through a task generating script, and establishes a corresponding relation between the task and a logic function of a corresponding task operation module;
and the task running module receives the tasks distributed by the task scheduling module and automatically executes preset task running script codes to process and analyze the tasks.
2. The system for scheduling of a trust analysis task of claim 1, wherein the analysis mirror repository module implements isolation of project and user permissions, further comprising master and slave replication logic for customizing user and system repository backups, capable of isolating sensitive operations of user and system repositories.
3. The system for scheduling of the student credit analysis task according to claim 1, wherein the mirror image intelligent pull module performs customized development on Docker and Harbor operations, provides a visual management interface, and supports viewing of the current local host mirror image condition.
4. The letter generation analysis task scheduling system of claim 3, wherein customized development of Docker and Harbor operations encapsulates both Docker self-execution command embedded mirror image intelligent pull modules and Harbor mirror image warehouse API interfaces embedded mirror image intelligent pull modules, and users obtain warehouse information in real time through the mirror image intelligent pull modules.
5. The credit generation analysis task scheduling system of claim 1, wherein the personalized configuration of the task scheduling module solves the phenomenon of computing resource waste or squashing in the credit generation analysis process by configuring the parameters of the battery and writing related service codes.
6. The letter generation analysis task scheduling system of claim 1, wherein the task running module automatically resolves letter generation analysis requirements transmitted by a webpage into parameters, packages the parameters into execution commands of a Docker to perform letter generation analysis tasks, and supports real-time monitoring of execution conditions of the tasks.
7. The system according to claim 6, wherein in the real-time monitoring of the execution, the task execution module integrates a battery monitoring tool, Flower, to visually check the execution and health of each raw credit analysis task; and/or the presence of a gas in the gas,
in the process of carrying out the credit generation analysis tasks by the Docker execution command, each credit generation analysis task establishes a specific temporary folder for receiving and analyzing the result output of the credit generation analysis task executed in the mirror image container, and storing the result into a file system after the analysis tasks are completely finished.
8. A method for building a credit generation analysis task scheduling system based on container technology according to any one of claims 1 to 7, wherein the method comprises the following steps:
the method comprises the following steps: installing a Docker engine on a host computer needing to run an analysis task, and starting the Docker engine;
step two: building an analysis mirror image warehouse module;
step three: compiling a system environment and a Dockerfile file of software, which depend on the letter analysis process, constructing an individualized version mirror image, and uploading the individualized version mirror image to a user mirror image warehouse;
step four: constructing a task scheduling module;
step five: and constructing a task operation module, compiling a visual webpage front-end interface and supporting the filling of analysis information.
9. The method for building a credit generation analysis task scheduling system based on container technology according to claim 8, wherein in the second step, the analysis mirror image warehouse module is built based on a Harbor, a docker-compound is used for starting a Harbor service, user authority and project authority of the Harbor warehouse are configured, a master-slave copy service is configured, and backup of an analysis mirror image is carried out; and/or the presence of a gas in the gas,
in the third step, the content of the user mirror image warehouse is automatically updated to the system mirror image warehouse, the deletion operation of the user mirror image warehouse is asynchronous with the system mirror image warehouse, and the system mirror image warehouse only has the authority of deletion operation by an administrator; and/or the presence of a gas in the gas,
the task scheduling module in the fourth step comprises a message middleware, a task execution unit and a task result storage unit; redis is integrated in the task scheduling system and is used as message middleware and is responsible for delivering and distributing task messages; the task execution unit is responsible for executing specific tasks and concurrently runs in distributed system nodes; the task result storage unit is responsible for storing the operation result of the task execution unit; the task scheduling system integrates a monitoring tool, namely a Flower, and monitors and presents the execution condition and the health condition of each letter generation analysis task in a visual mode; the task scheduling system carries out health check on the resource condition of each host system in the current cluster environment and selects a host with proper resource for task delivery; and/or the presence of a gas in the gas,
in the fifth step, the task operation module writes the analysis content in the analysis personnel and submits the task, and the task is automatically analyzed into a Docker execution command; the task scheduling system acquires a Docker execution command of an analysis task and automatically transmits the Docker execution command to the task scheduling module, and the task scheduling module intelligently selects a host with proper resources to perform credit generation analysis tasks according to the resource condition of the cluster system; after the letter generation analysis task is executed and a generated result is obtained, the task scheduling system stores the operation result of the execution unit, and the task operation module is responsible for collecting and storing a letter generation analysis result file.
10. A credit generation analysis task scheduling scheme based on the system of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011465097.5A CN112764894A (en) | 2020-12-14 | 2020-12-14 | Credit generation analysis task scheduling system based on container technology, and construction method and scheduling scheme thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011465097.5A CN112764894A (en) | 2020-12-14 | 2020-12-14 | Credit generation analysis task scheduling system based on container technology, and construction method and scheduling scheme thereof |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112764894A true CN112764894A (en) | 2021-05-07 |
Family
ID=75693807
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011465097.5A Pending CN112764894A (en) | 2020-12-14 | 2020-12-14 | Credit generation analysis task scheduling system based on container technology, and construction method and scheduling scheme thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112764894A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115268909A (en) * | 2022-07-23 | 2022-11-01 | 杭州沧浪健康管理有限公司 | Method, system and terminal for establishing and running construction task at web front end |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108427641A (en) * | 2018-01-29 | 2018-08-21 | 中国互联网络信息中心 | A kind of multi-task scheduling automated testing method and system based on Docker containers |
| US20190087220A1 (en) * | 2016-05-23 | 2019-03-21 | William Jason Turner | Hyperconverged system equipped with an orchestrator for installing and coordinating container pods on a cluster of container hosts |
| CN109614219A (en) * | 2018-10-19 | 2019-04-12 | 东莞理工学院 | A kind of condor duty mapping method of remote sensing image processing Docker cluster |
| CN110502212A (en) * | 2018-05-16 | 2019-11-26 | 南京慕测信息科技有限公司 | It is a kind of towards the multilingual online Development Support method of high concurrent |
-
2020
- 2020-12-14 CN CN202011465097.5A patent/CN112764894A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190087220A1 (en) * | 2016-05-23 | 2019-03-21 | William Jason Turner | Hyperconverged system equipped with an orchestrator for installing and coordinating container pods on a cluster of container hosts |
| CN108427641A (en) * | 2018-01-29 | 2018-08-21 | 中国互联网络信息中心 | A kind of multi-task scheduling automated testing method and system based on Docker containers |
| CN110502212A (en) * | 2018-05-16 | 2019-11-26 | 南京慕测信息科技有限公司 | It is a kind of towards the multilingual online Development Support method of high concurrent |
| CN109614219A (en) * | 2018-10-19 | 2019-04-12 | 东莞理工学院 | A kind of condor duty mapping method of remote sensing image processing Docker cluster |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115268909A (en) * | 2022-07-23 | 2022-11-01 | 杭州沧浪健康管理有限公司 | Method, system and terminal for establishing and running construction task at web front end |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11704224B2 (en) | Long running workflows for robotic process automation | |
| US20210117895A1 (en) | Systems and Methods for Cross-Platform Scheduling and Workload Automation | |
| CN101699393B (en) | Network-facing intelligent software interface dynamic generation method | |
| CN110196731B (en) | Operation and maintenance system, method and storage medium | |
| US20190018874A1 (en) | Database interface agent for a tenant-based upgrade system | |
| US10983824B2 (en) | Remotely monitoring and scheduling a data integration job | |
| US11110601B2 (en) | Scheduling robots for robotic process automation | |
| US20090164979A1 (en) | System landscape trace | |
| CN109840144B (en) | Information service scheduling method and system for cross-mechanism batch service request | |
| US20030192027A1 (en) | Software application development | |
| CN113434158B (en) | Custom management method, device, equipment and medium for big data component | |
| US9900212B2 (en) | Installation of an arbitrary server as an extension of a computing platform | |
| CN114064213B (en) | Quick arranging service method and system based on Kubernets container environment | |
| CN111258565A (en) | Method, system, server and storage medium for generating small program | |
| CN110874272A (en) | Resource allocation method and device, computer readable storage medium and electronic device | |
| CN113778486A (en) | Containerization processing method, device, medium and equipment for code pipeline | |
| CN112764894A (en) | Credit generation analysis task scheduling system based on container technology, and construction method and scheduling scheme thereof | |
| Maeno et al. | PanDA: Production and Distributed Analysis System | |
| CN111291106A (en) | Efficient flow arrangement method and system for ETL system | |
| CN115237547B (en) | Unified container cluster hosting system and method for non-invasive HPC computing cluster | |
| CN113961570A (en) | Real-time acquisition method applied to MYSQL BINLog change data | |
| WO2021126255A1 (en) | Querying development toolchain work items in batches | |
| US12008488B2 (en) | Systems and methods to manage sub-chart dependencies with directed acyclic graphs | |
| EP4386555A1 (en) | Report reexecution framework | |
| Fraile et al. | A scaffolding design framework for developing secure interoperability components in digital manufacturing platforms |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210507 |
|
| RJ01 | Rejection of invention patent application after publication |