CN116974716A

CN116974716A - Scheduling task issuing method and device, electronic equipment and storage medium

Info

Publication number: CN116974716A
Application number: CN202310121063.1A
Authority: CN
Inventors: 王巨宏; 余斐; 黄志斌; 王一男; 张贝; 刘炳贤; 谭用; 郭佳伟
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2023-01-31
Filing date: 2023-01-31
Publication date: 2023-10-31

Abstract

The embodiment of the application provides a method and a device for issuing a scheduling task, electronic equipment and a storage medium, and relates to the technical field of computers. The issuing method of the scheduling task can comprise the following steps: receiving a version issuing instruction aiming at a dispatching task, and sending the version issuing instruction to a Continuous Integration (CI) tool; receiving an upgrading task request sent by a CI tool; the upgrade task request is triggered based on the CI tool receiving a push success prompt returned by the first storage warehouse; responding to the upgrade task request, and sending an upgrade task to the version management client; the upgrade task is used for indicating the version management client to call a container arrangement engine cluster in the user server, acquiring a storage file from the first storage warehouse, and importing a code file of the scheduling task and a corresponding arrangement rule information file into a scheduling system of the user server based on the storage file so that the scheduling system triggers the execution of the scheduling task. The distribution of the scheduling task of the privately-owned user server can be realized.

Description

Scheduling task issuing method and device, electronic equipment and storage medium

Technical Field

The present application relates to the field of computer technologies, and in particular, to a method and apparatus for issuing a scheduling task, an electronic device, and a storage medium.

Background

Task scheduling refers to the process of a system to execute a task at a specific moment in time in order to automatically complete a specific task. More manpower can be liberated by the task scheduling, and the system can automatically execute the task. It is commonly used in big data offline computing tasks. Release of the version of the scheduled task requires release of the code and the arrangement rule of the scheduled task to the production environment.

In public cloud network environment, the release system can call the application programming interface of the basic component through the network request, so as to realize the service update of the production environment and achieve the purpose of release of the version of the scheduling task. However, in the privatized environment, the dispatching system and the publishing system in the privatized user environment are isolated from each other by the network due to network security, so that privatized publishing of the dispatching task cannot be realized.

Disclosure of Invention

The embodiment of the application aims to provide a method and a device for issuing a scheduling task and electronic equipment, and the technical scheme provided by the embodiment of the application is as follows:

In one aspect, an embodiment of the present application provides a method for scheduling task publication, which is applied to a publication system, and the method includes:

receiving a version issuing instruction aiming at a dispatching task, and sending the version issuing instruction to a Continuous Integration (CI) tool;

the version issuing instruction is used for indicating the CI tool to acquire a corresponding storage file of the scheduling task through the created pipeline and pushing the storage file into the first storage warehouse; the storage file is generated based on the code file of the scheduling task and the corresponding arrangement rule information file;

receiving an upgrading task request sent by a CI tool; the upgrade task request is triggered based on the CI tool receiving a push success prompt returned by the first storage warehouse;

responding to the upgrade task request, and sending an upgrade task to the version management client;

the upgrade task is used for instructing the version management client to execute the following steps:

calling a container arrangement engine cluster to acquire a storage file from a first storage warehouse;

based on the storage file, importing a code file of the scheduling task and a corresponding arrangement rule information file into a scheduling system so that the scheduling system triggers the execution of the scheduling task; the dispatch system is network isolated from the publication system.

In some possible embodiments, the method further comprises:

acquiring configuration information sent by an operation terminal and aiming at a user server;

receiving a pairing request of a version management client;

responding to the pairing request, checking the validity of the version management client based on the configuration information, and if the verification is passed, setting the state of the version management client as an executable state;

sending the upgrade task to the version management client, comprising:

and acquiring the state of the version management client, and if the state of the version management client is an executable state, sending an upgrading task to the version management client.

In some possible implementations, the pipeline is created based on the following:

receiving a creation instruction aiming at a pipeline and sent by an operation terminal;

responding to the creation instruction, and generating creation parameters of the pipeline based on the configuration information and a preset initial pipeline;

the creation parameters are sent to the CI tool to cause the CI tool to create a pipeline based on the creation parameters.

In some possible embodiments, before receiving the version issue instruction for the scheduled task and sending the version issue instruction to the CI tool, the method further includes:

if the CI tool detects that the code file aiming at the scheduling task in the second storage warehouse accords with the preset condition through the pipeline, testing the code file and the arrangement rule information file of the scheduling task through the testing system;

The code file of the scheduling task is sent to the second storage warehouse by the development terminal; the scheduling task arranging rule information file is sent to the testing system by the development terminal;

receiving a version issue instruction for a scheduled task, and sending the version issue instruction to a CI tool, comprising:

and receiving a version release instruction aiming at the dispatching task, and if the test result accords with the preset result, sending the version release instruction to the CI tool.

In some possible implementations, the storage file includes a container image;

obtaining a corresponding storage file of the scheduling task, including:

acquiring a code file of a scheduling task from a second storage warehouse, and deriving an arrangement rule information file from a test system;

and packaging the code file and the corresponding arrangement rule information file to generate a container mirror image.

In one aspect, an embodiment of the present application provides a method for scheduling the release of tasks, applied to a CI tool, where the method includes:

receiving a version release instruction aiming at a scheduling task and sent by a release system;

responding to the version issuing instruction, acquiring a corresponding storage file of the scheduling task through the created pipeline, and pushing the storage file into a first storage warehouse; the storage file is generated based on the code file of the scheduling task and the corresponding arrangement rule information file;

Receiving a pushing success prompt returned by the first storage warehouse, and responding to the pushing success prompt to send an upgrading task request to the issuing system;

the upgrade task request is used for indicating the issuing system to send an upgrade task to the version management client; the upgrade task is used for indicating the version management client to call the container arrangement engine cluster, acquiring a storage file from the first storage warehouse, and importing a code file of the scheduling task and a corresponding arrangement rule information file into the scheduling system based on the storage file so that the scheduling system triggers the execution of the scheduling task; the dispatch system is network isolated from the publication system.

In one aspect, an embodiment of the present application provides a method for scheduling task publication, applied to a version management client, where the method includes:

receiving an upgrade task sent by a release system; the upgrade task issuing system is triggered based on an upgrade task request sent by the CI tool; the upgrade task request is triggered by the CI tool receiving a push success prompt for the storage file returned by the first storage warehouse; the storage file is a version release instruction sent by the CI tool based on the release system, and is obtained and pushed to the first storage warehouse through the created pipeline; the storage file is generated based on the code file of the scheduling task and the corresponding arrangement rule information file; the version management client is isolated from the release system by a network;

and importing the code file of the scheduling task and the corresponding arrangement rule information file into a scheduling system based on the storage file so that the scheduling system triggers the execution of the scheduling task.

On the other hand, the embodiment of the application provides a device for issuing scheduling tasks, which comprises:

the first receiving module is used for receiving a version release instruction aiming at the dispatching task and sending the version release instruction to the continuous integration CI tool;

the second receiving module is used for receiving an upgrading task request sent by the CI tool; the upgrade task request is triggered based on the CI tool receiving a push success prompt returned by the first storage warehouse;

the first sending module is used for responding to the upgrade task request and sending the upgrade task to the version management client;

In some possible embodiments, the apparatus further comprises a verification module for:

receiving configuration information sent by an operation terminal and aiming at a user server;

receiving a pairing request of a user server; the user server is provided with a component for executing the scheduling task through an operation terminal, and is provided with a version management client through the operation terminal;

the first sending module is specifically configured to, when sending an upgrade task to the version management client:

generating, in response to the create instruction, a create parameter of the pipeline based on the configuration information and the initial pipeline;

In some possible embodiments, the apparatus further comprises a test module for:

the method further comprises the steps of before receiving the version release instruction aiming at the dispatching task and sending the version release instruction to the CI tool:

the first receiving module is specifically configured to, when receiving a version issue instruction for a scheduled task and sending the version issue instruction to the CI tool:

the third receiving module is used for receiving a version release instruction aiming at the scheduling task and sent by the release system;

the second acquisition module is used for responding to the version issuing instruction, acquiring a storage file corresponding to the scheduling task through the created assembly line and pushing the storage file into the first storage warehouse; the storage file is generated based on the code file of the scheduling task and the corresponding arrangement rule information file;

the fourth receiving module is used for receiving a pushing success prompt returned by the first storage warehouse and sending an upgrading task request to the issuing system in response to the pushing success prompt;

the fifth receiving module is used for receiving the upgrade task sent by the release system; the upgrade task issuing system is triggered based on an upgrade task request sent by the CI tool; the upgrade task request is triggered by the CI tool receiving a push success prompt for the storage file returned by the first storage warehouse; the storage file is a version release instruction sent by the CI tool based on the release system, and is obtained and pushed to the first storage warehouse through the created pipeline; the storage file is generated based on the code file of the scheduling task and the corresponding arrangement rule information file;

the calling module is used for calling a container arrangement engine cluster in the user server and acquiring a storage file from the first storage warehouse;

and the importing module is used for importing the code file of the scheduling task and the corresponding arrangement rule information file into the scheduling system of the user server based on the storage file so that the scheduling system triggers the execution of the scheduling task.

In another aspect, an embodiment of the present application further provides an electronic device, where the electronic device includes a memory and a processor, and the memory stores a computer program, and the processor executes the computer program to implement the method provided in any of the alternative embodiments of the present application.

In another aspect, embodiments of the present application also provide a computer readable storage medium having stored therein a computer program which, when executed by a processor, implements the method provided in any of the alternative embodiments of the present application.

In another aspect, embodiments of the present application also provide a computer program product comprising a computer program which, when executed by a processor, implements the method provided in any of the alternative embodiments of the present application.

The technical scheme provided by the embodiment of the application has the following beneficial effects:

when the release system receives the upgrade task request, the upgrade task version is sent to the version management client, the container arrangement engine cluster in the user server is called through the version management client, the storage file is obtained from the first storage warehouse, the code file of the scheduling task and the corresponding arrangement rule information file are imported into the scheduling system of the user server, and the problem that the network of the direct access scheduling system of the release system is limited is avoided, so that the release of the scheduling task is realized.

In addition, the code file and the corresponding arrangement rule information file of the scheduling task are stored in the first storage warehouse through the assembly line, when the scheduling task needs to be issued, the code file and the corresponding arrangement rule information file are obtained from the first storage warehouse and are imported into the scheduling system, and the issuing of the code and the arrangement rule can be automatically executed, so that the labor cost is effectively reduced, and the version issuing efficiency of the scheduling task is improved.

Furthermore, by presetting the initial pipeline, the creation parameters of the pipeline are generated according to the configuration information corresponding to different user servers and the initial pipeline, so that the corresponding pipeline is created, and the generation efficiency of the pipeline can be effectively improved.

Furthermore, by means of the containerization technology, all versions are container images, when the production environment has abnormal problems, version rollback can be rapidly realized, subsequent problems are avoided, and the effect of one-key rollback of a scheduling task is achieved.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that are required to be used in the description of the embodiments of the present application will be briefly described below.

FIG. 1 is a schematic illustration of an application environment for a method of publishing scheduled tasks provided in one example;

FIG. 2 is a schematic flow chart of a method for issuing a scheduling task according to an embodiment of the present application;

FIG. 3 is a schematic diagram of an interface for configuration information provided in one example of the present application;

FIG. 4 is a schematic diagram of a solution provided in one example of pairing a version management client with a publication system;

FIG. 5 is a schematic diagram of a scheme for creating a pipeline provided in one example of the application;

FIG. 6 is a schematic diagram of a version management client deployment scenario provided in one example of the present application;

FIG. 7 is a schematic diagram of a scheme for scheduling the publication of tasks provided in one example of the application;

FIG. 8 is a schematic diagram of a scheme for scheduling the publication of tasks provided in one example of the application;

FIG. 9 is a schematic diagram of a scheme provided in one example of the present application for scheduling the publication of tasks;

fig. 10 is a schematic structural diagram of a task scheduling issuing apparatus according to an embodiment of the present application;

fig. 11 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

Embodiments of the present application are described below with reference to the drawings in the present application. It should be understood that the embodiments described below with reference to the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application, and the technical solutions of the embodiments of the present application are not limited.

As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless expressly stated otherwise, as understood by those skilled in the art. It will be further understood that the terms "comprises" and "comprising," when used in this specification, specify the presence of stated features, information, data, steps, operations, elements, and/or components, but do not preclude the presence or addition of other features, information, data, steps, operations, elements, components, and/or groups thereof, all of which may be included in the present specification. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. The term "and/or" as used herein indicates that at least one of the items defined by the term, e.g., "a and/or B" may be implemented as "a", or as "B", or as "a and B". In describing a plurality of (two or more) items, if a relationship between the plurality of items is not explicitly defined, the plurality of items may refer to one, more or all of the plurality of items, for example, the description of "the parameter a includes A1, A2, A3" may be implemented such that the parameter a includes A1 or A2 or A3, and may also be implemented such that the parameter a includes at least two of three items of the parameters A1, A2, A3.

For a better description and understanding of the solution provided by the embodiments of the present application, first, some related technical terms involved in the embodiments of the present application will be described:

dolphin schedule: a distributed decentralization and easily-expanded visual DAG workflow task scheduling platform.

HDFS: designed as a distributed file system suitable for running on general purpose hardware.

Kubernetes, an open-source container orchestration engine, is used to automate deployment, scaling and management of containerized applications.

The release system comprises: after the software engineer develops the code, the code needs to be compiled and deployed to a production environment server through a release system, and the party can provide services to the outside. Developers usually only care about code implementation and function verification, and the deployment process is automatically completed by the release system, so that continuous integration and continuous deployment are achieved.

Task scheduling: refers to the process that a system performs a task at a specific moment in time in order to automatically complete a specific task. More manpower can be liberated by the task scheduling, and the system can automatically execute the task. There is a front-to-back relationship between tasks.

Privating deployment: typically referred to as deploying a third party application to the client's own server. Privately deployed is an external service mode commonly used by saas products.

Task arrangement: the task flows are connected in series according to a certain logic. The task comprises two elements, namely a front-back relation of the task and an operation rule of the task. For a work task, there may be different sub-tasks strung together, and some sub-tasks are performed in parallel or back and forth. The task itself also contains its own running rules such as start execution time, scheduling period, etc.

Deploying a pipeline: also known as continuous integrated pipeline, build pipeline, refers to an automated delivery process of software from a library of products to a user. Code submission, construction, deployment, testing, and release links are typically involved.

In the field of big data, the scheduling task provides services outside, not only comprises code development, but also needs to be subjected to task arrangement after the code deployment is completed. Task orchestration requires that the data developer determine according to specific requirements and cannot be done automatically. Because of this feature, big data scheduling tasks are more difficult to implement for sustained integration and sustained deployment. In the privatization scenario, each customer has its own exclusive operating environment. When releasing the version, the dispatching task needs to be released to the private running environment where the client is located. If there is no continuous integration and continuous deployment of scheduled tasks, then a developer is required to log into the proprietary environment of each customer when the version is released, manually maintaining the task orchestration. When the privately owned clients are more, it takes a lot of time for the developer to manually handle the release of the version, and also may introduce task anomalies due to human negligence. The system provides a method for dispatching task release, which is used for supporting continuous integration and continuous deployment of big data dispatching tasks.

When there is a new privately-placed customer order, the developer is also required to manually deploy and maintain the privately owned environment. The system provides a method for automatically applying continuous integration and continuous deployment of big data scheduling tasks to a new privately-used environment so as to improve the efficiency of generating the privately-used environment and releasing versions.

In a public cloud network environment, the publishing system can call an API (Application Programming Interface ) of a base component (kubernetes) through a network request, so that the service of the production environment is updated, and the purpose of version publishing is achieved. However, in the privatized environment, unlike public cloud network facilities, the privatized client environment is network isolated from the distribution system for network security. The publishing system cannot access the customer privatization environment. The system provides a method for realizing privately-distributed release of the scheduling task under the condition of not breaking network isolation.

In the current release of scheduling tasks, after project codes are submitted, a developer triggers the construction of a deployment pipeline, a docker (application container engine) mirror image is generated through packaging of a continuous integration tool, and the continuous integration tool calls an API of kubernetes to deploy the packaged mirror image into a produced kubernetes cluster. The entire process can be continuously and automatically performed.

For big data scheduling tasks, the complete operation of the tasks includes not only the code of the task, but also the task orchestration. The big data dispatching task normally runs and comprises a code for dispatching the task and a task arrangement rule.

For big data software projects, data developers need to manage task arrangement in a test environment after the codes are developed. After the release is completed, if only the release of the code is performed, task arrangement still needs to be managed in the production environment, and repetitive work exists. In a privatization scenario, the number of privatization environments is not fixed, and there are typically tens or even hundreds of privatization clients. If each privately-owned client needs a developer to manually manage task arrangement after the release is completed, the task arrangement is a great challenge for the developer.

And each privately-owned environment is independent, and service deployment of different environments cannot be realized through a single deployment pipeline. In addition, due to the consideration of network security, the publishing system cannot reach the client privateization environment, and a method for realizing version publishing by calling the component API of the client environment also has a problem.

Optionally, the data processing according to the embodiments of the present application may be implemented based on Cloud technology (Cloud technology), for example, the steps of creating a pipeline may be implemented using Cloud technology. Cloud technology refers to a hosting technology for unifying serial resources such as hardware, software, network and the like in a wide area network or a local area network to realize calculation, storage, processing and sharing of data. The cloud technology is based on the general names of network technology, information technology, integration technology, management platform technology, application technology and the like applied by the cloud computing business mode, can form a resource pool, and is flexible and convenient as required. Cloud computing technology will become an important support. Cloud computing refers to a delivery and use mode of an IT infrastructure, namely, obtaining required resources in an on-demand and easily-extensible manner through a network; generalized cloud computing refers to the delivery and usage patterns of services, meaning that the required services are obtained in an on-demand, easily scalable manner over a network. Such services may be IT, software, internet related, or other services. With the development of the internet, real-time data flow and diversification of connected devices, and the promotion of demands of search services, social networks, mobile commerce, open collaboration and the like, cloud computing is rapidly developed. Unlike the previous parallel distributed computing, the generation of cloud computing will promote the revolutionary transformation of the whole internet mode and enterprise management mode in concept.

The technical solution provided by the present application and the technical effects produced by the technical solution of the present application are described below by describing several alternative embodiments. It should be noted that the following embodiments may be referred to, or combined with each other, and the description will not be repeated for the same terms, similar features, similar implementation steps, and the like in different embodiments.

The method for issuing the scheduling task provided by the embodiment of the application can be executed by any computer equipment, and optionally can be executed by a server, wherein the server can be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server for providing cloud computing service.

Fig. 1 is an application environment schematic diagram of a method for issuing a scheduling task according to an embodiment of the present application.

The application environment may include a publishing terminal 101 and a user server 102, where the publishing terminal 101 includes a publishing system and a CI tool, and the user server 102 includes a version management client, a scheduling system, and a container orchestration engine cluster. Specifically, the release system receives a version release instruction aiming at a dispatching task and sends the version release instruction to the continuous integrated CI tool; the issuing system receives an upgrading task request sent by a CI tool; the issuing system responds to the upgrading task request and sends the upgrading task to the version management client; the version management client responds to the upgrading task and calls the container arrangement engine cluster to acquire a storage file from the first storage warehouse; based on the storage file, importing a code file of the scheduling task and a corresponding arrangement rule information file into a scheduling system so that the scheduling system triggers the execution of the scheduling task; the dispatch system is network isolated from the publication system.

In the application scenario, the release system and the CI tool are both set in the release terminal, and in other application scenarios, the release system may be set in the server, or the release system and the CI tool may be set in different devices, which is not limited to this application.

Those skilled in the art will appreciate that the server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server or a server cluster that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs (Content Delivery Network, content distribution networks), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal may be a smart phone (such as an Android mobile phone, an iOS mobile phone, etc.), a tablet computer, a notebook computer, a digital broadcast receiver, an MID (Mobile Internet Devices, mobile internet device), a PDA (personal digital assistant), a desktop computer, a smart home appliance, a vehicle-mounted terminal (such as a vehicle-mounted navigation terminal, a vehicle-mounted computer, etc.), a smart speaker, a smart watch, etc., and the terminal and the server may be directly or indirectly connected through wired or wireless communication manners, but are not limited thereto. The embodiment of the application can be applied to various scenes, including but not limited to cloud technology, artificial intelligence, intelligent transportation, auxiliary driving and the like. And in particular, the method can be determined based on actual application scene requirements, and is not limited herein.

A terminal (which may also be referred to as a user terminal or user device) may be, but is not limited to, a smart phone, tablet, notebook, desktop computer, smart voice interaction device (e.g., a smart speaker), wearable electronic device (e.g., a smart watch), vehicle-mounted terminal, smart home appliance (e.g., a smart television), AR/VR device, aircraft, etc. The embodiment of the application can be applied to various scenes, including but not limited to cloud technology, artificial intelligence, intelligent transportation, auxiliary driving and the like.

In some possible embodiments, a method for issuing a scheduling task is provided, which may be executed by an issuing system, where the issuing system may be disposed in a terminal or may be disposed in a server.

Fig. 2 is a schematic flow chart of a task scheduling issuing method according to an embodiment of the present application, taking an execution body as an example of an issuing system, where the task scheduling issuing method according to the present application may include the following steps:

step S201, receiving a version issue instruction for the scheduled task, and sending the version issue instruction to the persistent integration CI tool.

Specifically, the release system in the release system receives the version release instruction for the scheduled task, and sends the version release instruction to the CI (Continuous Integration, continuously integrated) tool in the CI tools.

In this embodiment, the publishing system is disposed in the publishing terminal, and the publishing system and the CI tool may be disposed in the same terminal, and in other embodiments, the publishing system or the CI tool may be disposed in the server, and the configuration of the specific publishing system and the CI tool is not limited in particular.

The version issuing instruction is used for instructing the CI tool to acquire a corresponding storage file of the scheduling task through the created pipeline and pushing the storage file into the first storage warehouse.

Specifically, the storage file is generated based on the code file of the scheduling task and the corresponding orchestration rule information file, and the storage file may include a container image.

Step S202, an upgrade task request sent by the CI tool is received.

Wherein the upgrade task request is triggered based on the CI tool receiving a push success prompt returned by the first storage repository.

Specifically, when the storage file is pushed to the first storage warehouse and the pushing is successful, the CI tool sends an upgrade task request to the publishing system.

Step S203, in response to the upgrade task request, the upgrade task is sent to the version management client.

The version management client may be a tool software package, for example, a tool software package written by java (a computer language), and runs on a client privately owned environment-specific server, i.e., in a user server. The version management client sends a network request to the release system through a specific network port and a signature, so that the problems of network limitation and network security of the release system in direct access to the privately-owned environment are avoided.

The version management client can be configured in a user server, and the user server further comprises a scheduling system and a container arrangement engine cluster.

Specifically, the version management client is isolated from the release system by a network, i.e. the release system cannot directly interact with the scheduling system.

Specifically, if the version management client receives the upgrade task, the container arrangement engine cluster in the user server is called, a storage file is obtained from the first storage warehouse, and based on the storage file, a code file of the scheduling task and a corresponding arrangement rule information file are imported into a scheduling system of the user server, and the scheduling system reissues the scheduling task.

In the above embodiment, when the publishing system receives the upgrade task request, the upgrade task version is sent to the version management client, and the version management client invokes the container arrangement engine cluster in the user server to obtain the storage file from the first storage warehouse, and the code file of the scheduling task and the corresponding arrangement rule information file are imported into the scheduling system of the user server, so that the problem that the network of the direct access scheduling system of the publishing system is limited is avoided, and the publishing of the scheduling task is realized.

The specific installation process of the version management client will be further described in connection with embodiments below.

In some possible embodiments, the method further comprises:

(1) And receiving configuration information for the user server sent by the operation terminal.

The configuration information may include, among other things, a service address, an API address, etc. of the user server, i.e. the user's privately owned environment.

As shown in fig. 3, in one example, the configuration information may include a corresponding service in the privately-owned environment, a privately-owned configuration name, an API address corresponding to the privately-owned production environment, a token corresponding to the privately-owned environment API, a corresponding privately-owned environment interface address, and so forth.

(2) And receiving a pairing request of the user server.

The user server is deployed with a component for executing the scheduling task through the operation terminal, and is provided with a version management client through the operation terminal.

(3) And responding to the pairing request, checking the validity of the version management client based on the configuration information, and if the verification is passed, setting the state of the version management client as an executable state.

Step S203 of sending the upgrade task to the version management client may include:

The state of the version management client is set to be an executable state, namely, the release process of the scheduling task can be executed.

Specifically, verifying the validity of the version management client based on the configuration information may include:

the version management client reads configuration information of the user server, and sends a pairing command to the issuing system;

the issuing system responds to the pairing command, matches the configuration information sent by the version management client with the configuration information sent by the operation terminal, checks whether the version management client is legal, if so, can execute the subsequent command, and sets the version management client to be in an operable state.

As shown in FIG. 4, in one example, the user server may also be referred to as a customer privatization environment. After the version management client is successfully installed, the distributed security credential information is used to send heartbeat connection to the release system according to a certain frequency. The publishing system receives the request of the client side of the privately-owned environment, namely the version management client side, and verifies the legitimacy of the version management client side through signature verification. The security credentials pass verification, i.e., the delegate publishing system and the client privately-owned environment pair successfully. The publishing system records the request time of each time of the client as an information source of whether the client is online or not. If the heartbeat request of the client is not received within a certain time, the publishing system triggers an alarm of the privatization environment.

Because of the network isolation, the distribution system cannot connect to the customer's proprietary environment. The version management client may be a tool software package written in Java (a computer language) into which proprietary environments, i.e. configuration information of the user server, are compiled. The version management client needs to be installed on a designated server, i.e. a user server, that has access to the public network. The version management client actively transmits a network request to the release system through a specific network port. The version management client and the release system form a client/server architecture, and the problem that the release system is limited to directly access the privately-arranged environment network is avoided. The client internal network does not need to be exposed to the public network, and secondly, the network security of the client and the server can be ensured due to mechanisms such as a specific server, a specific firewall, signature verification of the server and the like.

In the embodiment, the version management client is installed in the user server, so that the problem that the network of the dispatching system is limited in direct access of the issuing system is avoided, and the issuing of the dispatching task is realized. In addition, the privately-owned internal network of the user does not need to be exposed to the public network, and the network security can be ensured.

The deployment process of the pipeline will be further elucidated below in connection with the embodiments.

(1) Receiving a creation instruction aiming at a pipeline and sent by an operation terminal;

(2) Generating, in response to the create instruction, a create parameter of the pipeline based on the configuration information and the initial pipeline;

(3) The creation parameters are sent to the CI tool to cause the CI tool to create a pipeline based on the creation parameters.

Specifically, the initial pipeline may be a preset pipeline template, the publishing system may modify the initial pipeline according to configuration information of the user server, generate creation parameters for the pipeline, and then create the pipeline by the CI tool according to the creation parameters.

As shown in fig. 5, the publishing system acquires the configuration information of the user server, that is, acquires the configuration information of the privately owned environment, and generates the creation parameters according to the configuration information and the initial pipeline, and sends the creation parameters to the CI tool, that is, the creation pipeline command shown in the figure, and sends the creation parameters to the third-party CI platform, and the CI tool generates the pipeline, that is, the privately owned environment pipeline based on the creation parameters.

In the above embodiment, by presetting the initial pipeline, the creation parameters of the pipeline are generated according to the configuration information corresponding to different user servers and the initial pipeline, so that the corresponding pipeline is created, and the generation efficiency of the pipeline can be effectively improved.

The deployment procedure of the version management client of the present application will be explained below in connection with examples.

As shown in fig. 6, in one example, the deployment process of the version management client may include the steps of:

the operator terminal deploys components for performing scheduling tasks in the user server (i.e., the customer privatization environment shown in the figure);

if the operation terminal receives the deployment completion prompt sent by the user server, sending configuration information to the release system;

the operation terminal installs a version management client in a user server, and the version management client sends a pairing request to a release system;

the publishing system responds to the pairing request to pair with the version management client, and if the pairing is successful, the environment configuration is completed;

the operation terminal sends a command for creating the pipeline to an issuing system, and the issuing system generates pipeline parameters based on the configuration information and the initial pipeline and sends a creating request carrying the pipeline parameters to the CI tool;

The CI tool generates a pipeline based on the creation request.

In some possible embodiments, step S201 may further include, before receiving the version issue instruction for the scheduled task and sending the version issue instruction to the CI tool:

if the CI tool detects that the code file aiming at the scheduling task in the second storage warehouse accords with the preset condition through the pipeline, the code file and the arrangement rule information file of the scheduling task are tested through the testing system.

The code file of the scheduling task is sent to the second storage warehouse by the development terminal; the scheduling task arrangement rule information file is submitted to the test system by the development terminal.

Specifically, the test system may be set independently of the publishing terminal and the user server, and is used for testing the arrangement rule information file and the code file sent by the development terminal, and if the test is successful, the arrangement rule information file may be used for publishing the scheduling task.

Step S201 receives a version issue instruction for the scheduled task, and sends the version issue instruction to the CI tool, which may include:

The test result accords with a preset result, namely the arrangement rule information file can be used for issuing the scheduling task, namely the test is passed, and the issuing process of the scheduling task can be executed.

In some possible implementations, the storage file includes a container image (container image); the obtaining the corresponding storage file of the scheduling task may include:

Wherein the container image contains a packaged application, its dependencies, and process information it runs at start-up.

In the above embodiment, by means of the containerization technology, all versions are a container mirror image, when an abnormal problem occurs in the production environment, version rollback can be rapidly realized, subsequent problems are avoided, and the effect of one-key rollback of a scheduling task is achieved.

The publication process of the scheduled task of the present application will be described below in connection with examples.

As shown in fig. 7, in one example, the publishing process of the scheduled task of the present application may include the steps of:

The issuing system receives a version issuing instruction aiming at a dispatching task, namely a receiving issuing command shown in the figure, and sends the version issuing instruction to the CI tool, namely a sending pipeline executing request shown in the figure to a third party CI tool;

the CI tool acquires a corresponding storage file of the scheduling task through the created pipeline, and pushes the storage file to a first storage warehouse; the method comprises the steps that a trigger execution code is constructed, a dock mirror image is triggered to be generated, and the generated dock mirror image is pushed to a remote mirror image warehouse;

the release system sends an upgrade task to the version management client, and the version management client calls the container arrangement engine cluster to acquire a storage file from the first storage warehouse; based on the storage file, the code file of the scheduling task and the corresponding arrangement rule information file are imported into the scheduling system, so that the scheduling system triggers the execution of the scheduling task, namely Kubernetes triggers the execution of dockerstart (script), the code file is transferred to the HDFS (Hadoop Distributed File System, distributed file system), and the scheduling task is pulled from the HDFS to a user server where the scheduling system is located to execute the scheduling task when running.

The process of issuing the scheduled task shown in fig. 7, in combination with the test process before issuing, that is, before issuing the scheduled task, as shown in fig. 8, further includes:

and the version warehouse receives code information of the scheduling task sent by the development terminal, the test system tests the scheduling rule information file of the scheduling task, and if the test passes, the code file of the scheduling task is obtained from the version warehouse in the release process of the scheduling task, and the scheduling rule information file is derived from the test system.

The export module of FIG. 8 is configured to be executed by the deployment pipeline as a plug-in when the deployment pipeline triggers execution. The export module is used for packaging the task orchestration rules and codes into a container mirror image which can be issued. The dispatch system itself provides export APIs, and in the export module, the export APIs of the dispatch system are called to export the arrangement rules of all the dispatch tasks in the form of files, and then the export rules are packaged into a docker container image together with code files in a code base as release materials.

The import module in fig. 8 functions to deploy orchestration rules. After the mirror image package pushing to the remote warehouse is completed, the import module can be used as a plug-in to be executed by the deployment pipeline. The importing module is used for importing the packed arrangement rule information file into the scheduling system of the production environment by calling an importing API of the scheduling system of the production environment. Similar to code release, after the importing of the arrangement rule information file is completed, the previous scheduling task is removed through the scheduling system deletion API, so that the version replacement is completed.

After the release of the version is completed, the task codes and task arrangement of the production environment are complete copies of the test environment. If the test environment functions verify as passing, the same as the production environment is also a verified version. Other processes, including version rollback of scheduled tasks, refer to the general process of code release.

After the version image pushing is successful, the deployment pipeline submits the privately-owned environment image updating request to the issuing system by using the security certificate. The version management client requests the release system at regular time, and when version update exists in the release system, the release system issues a mirror image update command to the version management client. The version management client pulls the version image pushed before from the image warehouse through locally calling a Kubernetes image update command. The version image contains the logic of the import module, and after the pulling is completed, the container starts to execute the import module mentioned above.

The above-described distribution process of the scheduled tasks will be explained below from the point of view of interaction of the distribution system, the CI tool, and the scheduling system.

As shown in fig. 9, in an example, the method for issuing a scheduling task according to the present application may include:

the release system receives a version release instruction aiming at the dispatching task and sends the version release instruction to the continuous integration CI tool;

The CI tool acquires a corresponding storage file of the scheduling task through the created pipeline, and pushes the storage file to a first storage warehouse, namely a mirror image warehouse shown in the figure;

the CI tool receives a pushing success prompt returned by the storage warehouse, creates and sends an upgrading task request to the issuing system; namely, an image upgrading creation command shown in the figure and an image upgrading task request are created;

the issuing system responds to the upgrading task request and sends the upgrading task to the version management client; i.e. the image upgrade task is issued to the version management client as shown in the figure;

the version management client calls a container arrangement engine cluster to acquire a storage file from a first storage warehouse; that is, the image update command of the Kubernetes is executed, and the Kubernetes system pulls the issued version image from the image bin library;

the version management client side imports the code file of the scheduling task and the corresponding arrangement rule information file into the scheduling system based on the storage file so that the scheduling system triggers the execution of the scheduling task; i.e. running dockerstart as shown in the figure, executing the import module.

In some possible embodiments, there is provided a method for issuing a scheduled task, applied to a CI tool, the method including:

Specifically, when the version management client receives the upgrade task, a container arrangement engine cluster in the user server is called, a storage file is obtained from the first storage warehouse, a code file of the scheduling task and a corresponding arrangement rule information file are imported to a scheduling system of the user server based on the storage file, and then the scheduling system triggers the execution of the scheduling task.

The detailed description of the interaction process between the release system and the CI tool and the scheduling system using the release system as the execution body is omitted herein.

In some possible embodiments, a method for publishing a scheduling task is provided, applied to a version management client, and the method includes:

According to the method for issuing the scheduling task, when the issuing system receives the upgrading task request, the upgrading task version is sent to the version management client, the version management client calls the container arrangement engine cluster in the user server to acquire the storage file from the first storage warehouse, and the code file of the scheduling task and the corresponding arrangement rule information file are imported into the scheduling system of the user server, so that the problem that the network of the direct access scheduling system of the issuing system is limited is avoided, and the issuing of the scheduling task is realized.

As shown in fig. 10, in some possible embodiments, there is provided a publishing device for scheduling tasks, including:

the first receiving module 1001 is configured to receive a version issue instruction for a scheduled task, and send the version issue instruction to the persistent integrated CI tool;

a second receiving module 1002, configured to receive an upgrade task request sent by the CI tool; the upgrade task request is triggered based on the CI tool receiving a push success prompt returned by the first storage warehouse;

A first sending module 1003, configured to send an upgrade task to the version management client in response to the upgrade task request;

the first receiving module 1001, when receiving a version issue instruction for a scheduled task and sending the version issue instruction to the CI tool, is specifically configured to:

In some possible embodiments, there is provided a publishing device for scheduling tasks, the device comprising:

According to the dispatching task issuing device, when the issuing system receives the upgrading task request, the upgrading task version is sent to the version management client, the version management client calls the container arrangement engine cluster in the user server to obtain the storage file from the first storage warehouse, and the code file of the dispatching task and the corresponding arrangement rule information file are imported into the dispatching system of the user server, so that the problem that the network of the dispatching system is limited to be directly accessed by the issuing system is avoided, and the dispatching task is issued.

The device of the embodiment of the present application may perform the method provided by the embodiment of the present application, and its implementation principle is similar, and actions performed by each module in the device of the embodiment of the present application correspond to steps in the method of the embodiment of the present application, and detailed functional descriptions of each module of the device may be referred to the descriptions in the corresponding methods shown in the foregoing, which are not repeated herein.

An embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory, where the processor, when executing the computer program stored in the memory, may implement a method according to any of the alternative embodiments of the present application.

Fig. 11 is a schematic structural diagram of an electronic device, which may be a server or a user terminal, and may be used to implement the method provided in any embodiment of the present application, as shown in fig. 11, where the embodiment of the present application is applicable.

As shown in fig. 11, the electronic device 1100 may mainly include at least one processor 1101 (one is shown in fig. 11), a memory 1102, a communication module 1103, and an input/output interface 1104, and optionally, the components may be in communication with each other through a bus 1105. It should be noted that the structure of the electronic device 1100 shown in fig. 11 is merely schematic, and does not limit the electronic device to which the method provided in the embodiment of the present application is applicable.

The memory 1102 may be used to store an operating system, application programs, and the like, which may include computer programs that implement the methods of the embodiments of the present application when called by the processor 1101, and may include programs for implementing other functions or services. The Memory 1102 may be, but is not limited to, ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, RAM (Random Access Memory ) or other type of dynamic storage device that can store information and computer programs, EEPROM (Electrically Erasable Programmable Read Only Memory ), CD-ROM (Compact Disc Read Only Memory, compact disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.

The processor 1101 is connected to the memory 1102 through a bus 1105 and performs a corresponding function by calling an application program stored in the memory 1102. The processor 1101 may be a CPU (Central Processing Unit ), general purpose processor, DSP (Digital Signal Processor, data signal processor), ASIC (Application Specific Integrated Circuit ), FPGA (Field Programmable Gate Array, field programmable gate array) or other programmable logic device, transistor logic device, hardware components, or any combination thereof, that may implement or perform the various exemplary logic blocks, modules, and circuits described in connection with the present disclosure. The processor 1101 may also be a combination that performs computing functions, such as a combination comprising one or more microprocessors, a combination of a DSP and a microprocessor, or the like.

The electronic device 1100 may connect to a network through a communication module 1103 (which may include, but is not limited to, a component such as a network interface) to enable interaction of data, such as sending data to or receiving data from other devices (e.g., user terminals or servers, etc.) through communication of the network with the other devices. The communication module 1103 may include a wired network interface and/or a wireless network interface, etc., i.e., the communication module may include at least one of a wired communication module or a wireless communication module.

Electronic device 1100 may connect to desired input/output devices such as keyboards, display devices, etc. through input/output interface 1104, electronic device 110 itself may have a display device, and may also be coupled to other display devices through interface 1104. Optionally, a storage device, such as a hard disk, may be connected to the interface 1104, so that data in the electronic device 1100 may be stored in the storage device, or data in the storage device may be read, and data in the storage device may be stored in the memory 1102. It is to be appreciated that the input/output interface 1104 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to the input/output interface 1104 may be an integral part of the electronic device 1100, or may be an external device connected to the electronic device 1100 when needed.

Bus 1105 used to connect the components may include a path to transfer information between the components. Bus 1105 may be a PCI (Peripheral Component Interconnect, peripheral component interconnect standard) bus or EISA (Extended Industry Standard Architecture ) bus, or the like. The bus 1105 may be classified into an address bus, a data bus, a control bus, and the like according to functions.

Optionally, for the solution provided by the embodiment of the present application, the memory 1102 may be configured to store a computer program for executing the solution of the present application, and the processor 1101 is configured to implement the actions of the method or the apparatus provided by the embodiment of the present application when the processor 1101 runs the computer program.

Based on the same principle as the method provided by the embodiment of the present application, the embodiment of the present application provides a computer readable storage medium, where a computer program is stored, where the computer program can implement the corresponding content of the foregoing method embodiment when executed by a processor.

Embodiments of the present application also provide a computer program product comprising a computer program which, when executed by a processor, implements the respective aspects of the method embodiments described above.

It should be noted that the terms "first," "second," "third," "fourth," "1," "2," and the like in the description and claims of the present application and in the above figures, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate, such that the embodiments of the application described herein may be implemented in other sequences than those illustrated or otherwise described.

It should be understood that, although various operation steps are indicated by arrows in the flowcharts of the embodiments of the present application, the order in which these steps are implemented is not limited to the order indicated by the arrows. In some implementations of embodiments of the application, the implementation steps in the flowcharts may be performed in other orders as desired, unless explicitly stated herein. Furthermore, some or all of the steps in the flowcharts may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or phases may be performed at the same time, or each of these sub-steps or phases may be performed at different times, respectively. In the case of different execution time, the execution sequence of the sub-steps or stages can be flexibly configured according to the requirement, which is not limited by the embodiment of the present application.

The foregoing is merely an optional implementation manner of some of the implementation scenarios of the present application, and it should be noted that, for those skilled in the art, other similar implementation manners based on the technical ideas of the present application are adopted without departing from the technical ideas of the scheme of the present application, and the implementation manner is also within the protection scope of the embodiments of the present application.

Claims

1. A publication method for scheduling tasks, applied to a publication system, said method comprising:

the CI tool is used for acquiring a corresponding storage file of a scheduling task through the created pipeline and pushing the storage file into a first storage warehouse; the storage file is generated based on the code file of the scheduling task and the corresponding arrangement rule information file;

receiving an upgrade task request sent by the CI tool; the upgrade task request is triggered based on the CI tool receiving a push success prompt returned by the first storage warehouse;

responding to the upgrade task request, and sending an upgrade task to a version management client;

invoking a container orchestration engine cluster to obtain the storage file from the first storage warehouse;

importing the code file of the scheduling task and the corresponding arrangement rule information file into a scheduling system based on the storage file so that the scheduling system triggers the execution of the scheduling task; the dispatch system is network isolated from the publication system.

2. The method according to claim 1, wherein the method further comprises:

receiving a pairing request of a version management client;

the sending the upgrade task to the version management client side includes:

and acquiring the state of the version management client, and if the state of the version management client is an operable state, sending an upgrade task to the version management client.

3. The method of claim 2, wherein the pipeline is created based on:

receiving a creation instruction for a pipeline sent by the operation terminal;

responding to the creation instruction, and generating creation parameters of a pipeline based on the configuration information and a preset initial pipeline;

the creation parameters are sent to a CI tool to cause the CI tool to create the pipeline based on the creation parameters.

4. The method of claim 1, wherein the receiving a version issue instruction for a scheduled task, before sending the version issue instruction to a CI tool, further comprises:

If the CI tool detects that the code file of the scheduling task in the second storage warehouse accords with the preset condition through the pipeline, the code file of the scheduling task and the arrangement rule information file are tested through a testing system;

the code file of the scheduling task is sent to the second storage warehouse by the development terminal; the scheduling task arranging rule information file is sent to the test system by the development terminal;

the receiving a version issue instruction for a scheduled task, and sending the version issue instruction to a CI tool, includes:

5. The method of claim 1, wherein the storage file comprises a container image;

the obtaining the corresponding storage file of the scheduling task comprises the following steps:

acquiring a code file of a scheduling task from the second storage warehouse, and deriving the arrangement rule information file from the test system;

and packaging the code file and the corresponding arrangement rule information file to generate the container mirror image.

6. A method of publishing a scheduled task, characterized by being applied to a CI tool, the method comprising:

responding to the version issuing instruction, acquiring a storage file corresponding to the scheduling task through the created pipeline, and pushing the storage file into a first storage warehouse; the storage file is generated based on the code file of the scheduling task and the corresponding arrangement rule information file;

the upgrade task request is used for indicating the issuing system to send an upgrade task to the version management client; the upgrade task is used for indicating the version management client to call a container arranging engine cluster, acquiring the storage file from the first storage warehouse, and importing a code file of the scheduling task and a corresponding arranging rule information file into a scheduling system based on the storage file so that the scheduling system triggers the execution of the scheduling task; the dispatch system is network isolated from the publication system.

7. A method for distributing scheduling tasks, applied to a version management client, the method comprising:

receiving an upgrade task sent by a release system; the issuing system is triggered based on an upgrading task request sent by a CI tool; the upgrading task request is triggered by the CI tool receiving a pushing success prompt for a storage file returned by a first storage warehouse; the storage file is a version release instruction sent by a CI tool based on a release system, and is obtained and pushed to the first storage warehouse through a created pipeline; the storage file is generated based on the code file of the scheduling task and the corresponding arrangement rule information file; the version management client is isolated from the release system by a network;

8. A publication apparatus for scheduling tasks, said apparatus comprising:

The first receiving module is used for receiving a version release instruction aiming at a dispatching task and sending the version release instruction to the continuous integrated CI tool;

the second receiving module is used for receiving the upgrading task request sent by the CI tool; the upgrade task request is triggered based on the CI tool receiving a push success prompt returned by the first storage warehouse;

the first sending module is used for responding to the upgrading task request and sending the upgrading task to the version management client;

9. A publication apparatus for scheduling tasks, said apparatus comprising:

the second acquisition module is used for responding to the version issuing instruction, acquiring a corresponding storage file of the scheduling task through the created pipeline and pushing the storage file into the first storage warehouse; the storage file is generated based on the code file of the scheduling task and the corresponding arrangement rule information file;

the fourth receiving module is used for receiving a pushing success prompt returned by the first storage warehouse, and sending an upgrading task request to the issuing system in response to the pushing success prompt;

10. A publication apparatus for scheduling tasks, said apparatus comprising:

the fifth receiving module is used for receiving the upgrade task sent by the release system; the issuing system is triggered based on an upgrading task request sent by a CI tool; the upgrading task request is triggered by the CI tool receiving a pushing success prompt for a storage file returned by a first storage warehouse; the storage file is a version release instruction sent by a CI tool based on a release system, and is obtained and pushed to the first storage warehouse through a created pipeline; the storage file is generated based on the code file of the scheduling task and the corresponding arrangement rule information file;

the calling module is used for calling a container arrangement engine cluster in the user server and acquiring the storage file from the first storage warehouse;

and the importing module is used for importing the code file of the scheduling task and the corresponding arrangement rule information file into a scheduling system of the user server based on the storage file so that the scheduling system triggers the execution of the scheduling task.

11. An electronic device comprising a memory having a computer program stored therein and a processor executing the computer program to implement the method of any of claims 1 to 7.

12. A computer-readable storage medium, characterized in that the storage medium has stored therein a computer program which, when executed by a processor, implements the method of any one of claims 1 to 7.

13. A computer program product, characterized in that the computer product comprises a computer program which, when executed by a processor, implements the method of any one of claims 1 to 7.