CN114860262A - Task-driven construction system oriented to offline environment - Google Patents
Task-driven construction system oriented to offline environment Download PDFInfo
- Publication number
- CN114860262A CN114860262A CN202210432482.2A CN202210432482A CN114860262A CN 114860262 A CN114860262 A CN 114860262A CN 202210432482 A CN202210432482 A CN 202210432482A CN 114860262 A CN114860262 A CN 114860262A
- Authority
- CN
- China
- Prior art keywords
- task
- deployment
- service
- database
- docker
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000010276 construction Methods 0.000 title claims abstract description 9
- 238000007726 management method Methods 0.000 claims description 17
- 238000013500 data storage Methods 0.000 claims description 6
- 238000012544 monitoring process Methods 0.000 claims description 5
- 238000013515 script Methods 0.000 abstract description 9
- 238000004458 analytical method Methods 0.000 abstract description 2
- 238000000034 method Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 3
- 238000012423 maintenance Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000005856 abnormality Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Stored Programmes (AREA)
Abstract
The invention belongs to the technical field of computer data service, and particularly relates to a task driving construction system facing an offline environment. By integrating the anchor scheme and the Docker scheme, the core code logic of the anchor is integrated, and the script driving model of the anchor is rewritten into a class-based driving model, so that the unified structure of the task is standardized, and the function of directly calling and executing the task in the program is realized; supporting event-based analysis and triggering corresponding tasks; the error log is optimized and the errors can be read directly in the database. The system native supports docker, and the support programming specifies services suitable for docker operation and services not suitable for docker operation, wherein the former uses docker virtual machines as execution environments, and the latter uses physical machines as execution environments.
Description
Technical Field
The invention belongs to the technical field of computer data service, and particularly relates to a task driving construction system facing an offline environment.
Background
The program needs to run normally on the server, and firstly, the configuration is the basic running environment configuration of the program, which involves the customized configuration according to the target machine architecture, the used operating system environment, the target network environment, the dependent package of the program and the like. Manually performing environment configuration is often tedious, especially for multi-machine deployments.
A typical scenario is some integrated cloud service system, which contains hundreds of services, has complex dependencies, and each service requires several profile generations. For system reliability, the system needs to be deployed over many places and different servers.
The infrastructure is a traditional solution to this scenario. The infrastructure is an IT automation tool, based on SSH, and through the infrastructure script, the complicated manual deployment logic can be automatically executed, so that the deployment process is simplified, and the deployment efficiency is improved. Meanwhile, the infrastructure supports multi-machine deployment, and the same infrastructure script can be executed on a plurality of machines to construct a deployment system.
Docker is another approach that utilizes resource separation mechanisms in the Linux kernel, such as cgroups, and the Linux kernel namespaces (namespaces), to create independent containers (containers). This can be done in a single Linux entity, avoiding the additional burden of booting a virtual machine. The Linux kernel's support of namespaces completely isolates the view of applications in the working environment, including the trip tree, network, user ID, and mounted file system, while the kernel's cgroup provides resource isolation, including CPU, memory, block I/O, and network. Thus, the service can be separated from the host machine, and the service is not influenced by the machine architecture, the operating system and the environment of the host machine.
The practical application of the technology has some defects in the integrated cloud service system. The problem with the infrastructure scheme is: firstly, when a system to be deployed is too complex and ansible scripts are too many, a structured management mode does not exist, and the calling is very complex; secondly, exception and error can occur due to various reasons when the ansible is called, and the error is mixed with a common log and is difficult to track; third, the infrastructure itself is an independent program, and has system compatibility problems, which depend on many, and is difficult to deploy in an offline environment. Problem with the Docker scheme: the first Docker is used as a virtual machine-based scheme, and a program has performance loss; secondly, the method comprises the following steps: the characteristics and Bug of Docker itself cause additional complexity, that is, the program needs to be executed to not only concern itself but also concern about Docker-related problems, such as network and Volume-related problems. Thirdly, the method comprises the following steps: due to the characteristics of part of services, the exception can be caused in a Docker environment, and the method is not suitable for Docker-based operation and deployment.
The above schemes all have self advantages and expose certain problems, but in the prior art, the above schemes are not integrated in operation mode temporarily, and cannot be operated efficiently.
Disclosure of Invention
In order to overcome the problems in the prior art and meet the needs of the business itself, the present invention aims to provide an event-based task-driven service construction system oriented to an offline environment. The specific technical scheme is as follows:
s1, deploying various services for normal operation of the system, including management service, authority service, external interface service, system monitoring service, server environment statistics service, search service and data storage service; a python query system which runs at a fixed time and is used for querying the occupation conditions of resources such as a system CPU, a memory and the like;
s2, adding a deployment task, operating codes under a local path and opening a port through a folder deployment service of the specified server;
s3, sending a deployment task to a database;
s4, the asynchronous deployment system queries the tasks to be executed in the database at the frequency of once per second; if the task to be executed is to be executed, the deployment management system analyzes the task and sends the parameters to the component corresponding to the task to be executed, and the component defines codes in advance and can automatically complete the work;
s5, the deployment management system records the execution result, and if the execution result is successful, the deployment management system marks that the task is successfully executed; if so, the task failure is noted and a detailed error log is logged to the database.
The plurality of services in the S1 are composed of a plurality of different Java services, including an elastic search service, a redis service, a kafka service, and a Java springboot extension-based server service.
And the step S2, wherein each task in the added deployment tasks is composed of a task name, a task state, a task execution parameter, a task execution time and a task executor.
And the deployment task sending tool is called to send the tasks to the database, and the deployment system can automatically asynchronously read and execute the deployment tasks from the database.
By integrating the anchor scheme and the Docker scheme, the core code logic of the anchor is integrated, and the script driving model of the anchor is rewritten into a class-based driving model, so that the unified structure of the task is standardized, and the function of directly calling and executing the task in the program is realized; supporting event-based analysis and triggering corresponding tasks; the error log is optimized and the errors can be read directly in the database. The system native supports docker, and the support programming specifies services suitable for docker operation and services not suitable for docker operation, wherein the former uses docker virtual machines as execution environments, and the latter uses physical machines as execution environments. Meanwhile, according to the requirements of the existing service, an interactive webpage is required to be provided, and the task is submitted through the webpage. And requires that the system be capable of normal use in an offline environment.
Drawings
FIG. 1 is a schematic diagram of task submission;
FIG. 2 is a schematic diagram of the execution logic;
FIG. 3 is a flow chart illustrating the implementation of the present invention.
Detailed Description
A task-driven construction system oriented to an offline environment comprises the following steps:
s1, deploying various services for normal operation of the system, including management service, authority service, external interface service, system monitoring service, server environment statistics service, search service and data storage service; a python query system which runs at a fixed time and is used for querying the occupation conditions of resources such as a system CPU, a memory and the like;
s2, adding a deployment task, operating codes under a local path and opening a port through a folder deployment service of the specified server;
s3, sending a deployment task to a database;
s4, the asynchronous deployment system queries the tasks to be executed in the database at the frequency of once per second; if the task to be executed is to be executed, the deployment management system analyzes the task and sends the parameters to the component corresponding to the task to be executed, and the component defines codes in advance and can automatically complete the work;
s5, the deployment management system records the execution result, and if the execution result is successful, the deployment management system marks that the task is successfully executed; if so, the task failure is noted and a detailed error log is recorded to the database.
The plurality of services in the S1 are composed of a plurality of different Java services, including an elastic search service, a redis service, a kafka service, and a Java springboot extension-based server service.
And the step S2, wherein each task in the added deployment tasks is composed of a task name, a task state, a task execution parameter, a task execution time and a task executor.
And the deployment task sending tool is called to send the tasks to the database, and the deployment system can automatically asynchronously read and execute the deployment tasks from the database.
In the task submission framework of the deployment system in fig. 1, the deployment system will periodically poll the database or the message queue to determine whether there is a new deployment task. The deployment tasks may be submitted from a program, script, or web page. Fig. 2 shows the logic executed by the deployment system, and after the deployment task is read, the offline packet or docker container required by the construction system is distributed to the target machine through the combination scheme of the infrastructure architecture + SSH architecture, the deployment task is executed, and the deployment result is written back to the database.
The invention relates to an event-driven deployment framework which supports multi-machine multi-task simultaneous deployment, the framework is currently applied to a deployment project of an integrated cloud service platform system, the system consists of a plurality of different Java services, and the system mainly comprises a system management service, an authority service, an external interface service, a system monitoring service, a server environment statistical service, a search service, a data storage service and the like. Each service is a separate computer program that runs on a server, responds to user requests, and performs different operations: for example, the system management service can increase and decrease the service, and can be opened and closed; the authority service is used for increasing and decreasing the authority of the user; the external interface service can provide a standard HTTP-based interface and respond to user requests; the system monitoring service can detect the current state of the system, the number of requests, the load, etc. In order to prevent the stability of the whole system from being affected by local power failure, network disconnection and the like of one or more servers, basic environments for program operation need to be established on a plurality of machines distributed in different regions, and at present, the machines of the system are distributed in a plurality of regions such as south China, North China, south China and the like. And because the system has dependencies, such as an external interface service relying on a data storage service, an authority service relying on a system management service, and the like, it is necessary to specify the order of service deployment, such as that the data storage service should be deployed first, and then the system management service, and the like. Meanwhile, in order to complete the deployment as soon as possible, a plurality of machines are required to start the deployment at the same time; the asynchronous system deployment framework is an independent program, interacts with other programs through events, and is low in coupling degree and low in reconstruction and upgrade cost. Taking an integrated cloud service platform as an example, in practice, adapting to a new deployment system requires simply modifying source codes, mainly adding required tasks, wherein each task is a structured class defined by a task name, a task state, task execution parameters, task execution time and a task executor, and a class code is about ten lines. The codes are compiled into the conventional technology in the field, for example, in an integrated cloud service platform instance, a certificate is added to generate a task class, the codes generate the task class, a service is constructed into the task class, the service is deployed into the task class, and the like, then a deployment task sending tool is called to send a task to a database, and a deployment system can automatically and asynchronously read and execute the deployment task from the database. Compared with the prior system, in the old system, if a task needs to be added, an interface needs to be added, a deployment shell script is written, and an external infrastructure is called in the script for deployment. There are up to hundreds of lines of redundant code among these, and this is only a task; deeply integrating the infrastructure core code, packaging an interface, performing the class infrastructure code in a modularized manner, and capturing the exception of the infrastructure in the deployment process. The deployment system is characterized in that the source code of the deployment system is transplanted based on an anchor execution engine, all the modules of the anchor are supported, only class definition needs to be used for replacing script statements of the anchor, an intermediate layer is added on the basis of the anchor core engine, the work of the anchor core engine is abstracted by the intermediate layer, an interface is added on the original code flow, the interface can be triggered in three stages before, during and after the flow execution, and whether a task needs to be executed in the stage or not or whether the execution is abnormal or not is judged, so that errors and abnormality of each stage are captured. For example, in the scenario of the integrated cloud service, the upload task may fail due to reasons that an opposite-end server port is not opened, a connection cannot be established, a certificate check fails, or an opposite-end server space is insufficient, where the first three scenarios throw an exception during the execution of the process, and the last scenario throws an exception after the execution of the process is finished. After the abnormity and the error are detected, the data can be written into a database, so that the data can be directly displayed on a page, and operation and maintenance personnel can be reminded through mails or short messages, so that the operation and maintenance personnel can conveniently master the deployment process; and accurately recording the execution condition of the deployment flow, and supporting the timed execution of the deployment task. The deployment system provides a visual page, and can know the task being deployed and the task deployment progress and situation in real time through a digital large screen. When the abnormity occurs, the operation and maintenance personnel can be notified through mails and short messages. In addition, the function of executing the deployment task at regular time is provided on the page, and the function is used for reserving the task to be executed at a specific time, for example, in an integrated cloud service platform system, service system self-checking is reserved in the morning. For programs suitable for docker operation, the deployment system native support executes docker containers, and is used for simplifying task and complexity of construction. The original gradually executed deployment process is converted into a push docker container, and a docker command is executed. And without regard to the architecture and dependency of the system. At present, a deployment system turns a large number of services with low tool and performance requirements into docker execution.
Specific examples are exemplified below:
deploying various services for normal operation of the system, such as an elastic search service; a redis service; a kafka service; a server service based on java springboot expansion; and the python query system runs at regular time and is used for querying the resource occupation conditions of a system CPU, a memory and the like. Adding a deployment task, such as a Springboot service deployment task, as follows: { "targetIp": 127.0.0.1"," targetPath ":"/tmp "," source ":"/tmp/source "," port ": 3001}, indicating that the service needs to be deployed under the/tmp folder of 127.0.0.1 this server, using the code under the local/tmp/source path, and opening port 3001. Sending deployment tasks to the database asynchronous deployment system queries the database for tasks to be performed at a frequency of once per second. If the task is to be executed, the deployment system analyzes the task and sends the parameters to the component corresponding to the task to be executed. The component is a code defined in advance, and can automatically complete the work defined by the parameters. Like the above SpringBoot deployment task, the SpringBoot deployment task is sent to the SpringBoot deployment component, and the deployment component copies the path of the source field, here/tmp/source, and uploads the path to targetPath of targetIP, here 127.0.0.1/tmp path. The deployment system records the execution result, and if the execution result is successful, the deployment system marks that the task is successfully executed; if so, the task failure is noted and a detailed error log is recorded to the database. The database reads the execution condition of tasks in the database through a visualized deployment page, and then displays which tasks are deployed successfully, which tasks are failed and what the failure reason is in the page.
Claims (4)
1. A task-driven construction system oriented to an offline environment is characterized by comprising the following parts:
s1, deploying various services for normal operation of the system, including management service, authority service, external interface service, system monitoring service, server environment statistics service, search service and data storage service; a python query system which runs at a fixed time and is used for querying the occupation conditions of resources such as a system CPU, a memory and the like;
s2, adding a deployment task, operating codes under a local path and opening a port through a folder deployment service of the specified server;
s3, sending a deployment task to a database;
s4, the asynchronous deployment system queries the tasks to be executed in the database at the frequency of once per second; if the task to be executed is to be executed, the deployment management system analyzes the task and sends the parameters to the component corresponding to the task to be executed, and the component defines codes in advance and can automatically complete the work;
s5, the deployment management system records the execution result, and if the execution result is successful, the deployment management system marks that the task is successfully executed; if so, the task failure is noted and a detailed error log is recorded to the database.
2. The task-driven building system oriented to the offline environment of claim 1, wherein said plurality of services in S1 are composed of a plurality of different Java services, including an elastic search service, a redis service, a kafka service, and a Java springboot extension-based server service.
3. The task-driven building system for an offline environment according to claim 1, wherein each task in the S2 incremental deployment task is composed of a task name, a task state, a task execution parameter, a task execution time, and a task executor.
4. The task-driven building system for an offline environment as recited in claim 1, wherein said deployment task sending tool is invoked to send a task to the database, and the deployment system will automatically asynchronously read and execute the deployment task from the database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210432482.2A CN114860262A (en) | 2022-04-24 | 2022-04-24 | Task-driven construction system oriented to offline environment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210432482.2A CN114860262A (en) | 2022-04-24 | 2022-04-24 | Task-driven construction system oriented to offline environment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114860262A true CN114860262A (en) | 2022-08-05 |
Family
ID=82633761
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210432482.2A Pending CN114860262A (en) | 2022-04-24 | 2022-04-24 | Task-driven construction system oriented to offline environment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114860262A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180239646A1 (en) * | 2014-12-12 | 2018-08-23 | Nec Corporation | Information processing device, information processing system, task processing method, and storage medium for storing program |
| CN110704164A (en) * | 2019-09-30 | 2020-01-17 | 珠海市新德汇信息技术有限公司 | Cloud native application platform construction method based on Kubernetes technology |
| CN112181592A (en) * | 2020-09-25 | 2021-01-05 | 上海聪充网络科技有限公司 | Method for lightweight deployment of distributed system in private cloud environment |
| CN112328361A (en) * | 2020-11-02 | 2021-02-05 | 杭州当虹科技股份有限公司 | Method for rapidly deploying kubenetes cluster |
-
2022
- 2022-04-24 CN CN202210432482.2A patent/CN114860262A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180239646A1 (en) * | 2014-12-12 | 2018-08-23 | Nec Corporation | Information processing device, information processing system, task processing method, and storage medium for storing program |
| CN110704164A (en) * | 2019-09-30 | 2020-01-17 | 珠海市新德汇信息技术有限公司 | Cloud native application platform construction method based on Kubernetes technology |
| CN112181592A (en) * | 2020-09-25 | 2021-01-05 | 上海聪充网络科技有限公司 | Method for lightweight deployment of distributed system in private cloud environment |
| CN112328361A (en) * | 2020-11-02 | 2021-02-05 | 杭州当虹科技股份有限公司 | Method for rapidly deploying kubenetes cluster |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110471831B (en) | Automatic method and device for compatibility test | |
| US7694181B2 (en) | Automated software testing framework | |
| US7392148B2 (en) | Heterogeneous multipath path network test system | |
| US7337435B2 (en) | Efficient configuration data migration technique | |
| CN112084008B (en) | Method for rapidly deploying cloud pipe system based on container technology | |
| CN111324599B (en) | Block chain experiment system and management method | |
| US7934199B2 (en) | Automated operation of IT resources with multiple choice configuration | |
| CN115686540A (en) | RPA control method and system based on Hongmong system | |
| CN112000567A (en) | Regulation and control software test service method based on cloud platform | |
| CN114860262A (en) | Task-driven construction system oriented to offline environment | |
| CN112596750B (en) | Application testing method and device, electronic equipment and computer readable storage medium | |
| CN112256554B (en) | Method and equipment for testing based on scene test cases | |
| Na et al. | Extension and application based on robot testing framework | |
| CN113986714A (en) | Automatic continuous testing method and device based on containerization | |
| CN113961395A (en) | Plug and play method of chip protection device and control device | |
| US20210019186A1 (en) | Information processing system, information processing apparatus, and method of controlling an information processing system | |
| CN111338871A (en) | Distributed file system Qzone high availability test method, system, equipment and storage medium | |
| CN116931965B (en) | Integrated stream processing method, device, electronic equipment and storage medium | |
| CN118503002B (en) | Operation and maintenance system based on cloud native application | |
| CN109348496B (en) | Laboratory base station fault simulation system and method | |
| CN118245346A (en) | Automatic operation and maintenance management system and method oriented to operating system | |
| Royon et al. | A Survey of Unix Init Schemes | |
| CN117008925A (en) | Application deployment management method, device, equipment and computer storage medium | |
| CN117349137A (en) | Automatic testing method, device, equipment and storage medium of distributed system | |
| CN118467217A (en) | Information recording method, apparatus, electronic device, and computer-readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |