CN114443292B - Client management method, device and system based on workflow task scheduling system - Google Patents
Client management method, device and system based on workflow task scheduling system Download PDFInfo
- Publication number
- CN114443292B CN114443292B CN202210044858.2A CN202210044858A CN114443292B CN 114443292 B CN114443292 B CN 114443292B CN 202210044858 A CN202210044858 A CN 202210044858A CN 114443292 B CN114443292 B CN 114443292B
- Authority
- CN
- China
- Prior art keywords
- client
- task
- task execution
- command
- execution end
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000007726 management method Methods 0.000 title claims abstract description 46
- 238000000034 method Methods 0.000 claims abstract description 22
- 238000012795 verification Methods 0.000 claims abstract description 17
- 230000006837 decompression Effects 0.000 claims abstract description 7
- 238000009434 installation Methods 0.000 claims description 123
- 238000004806 packaging method and process Methods 0.000 claims description 12
- 230000003993 interaction Effects 0.000 claims description 8
- 241001481833 Coryphaena hippurus Species 0.000 description 19
- 238000010586 diagram Methods 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 238000013461 design Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The application belongs to the technical field of distributed system management, and particularly provides a client management method, device and system based on a workflow task scheduling system, wherein the method comprises the following steps: judging whether a client is installed on a task execution end according to the environment variable of the system, and reporting and displaying a judging result; when the client is not installed on the task execution end, the client is installed in a mode of downloading a package decompression configuration environment variable or in a yum source mode; checking whether a client on a task execution end meets task operation or not; and when the verification finds that the client does not meet the condition, replacing the task execution end group or installing the client for the task execution end which does not meet the condition. In the aspect of workflow execution, before execution, the workflow is checked according to the workbench condition in the workbench group, so that the task is prevented from failing due to the fact that the client is not installed.
Description
Technical Field
The application relates to the technical field of distributed system management, in particular to a client management method, device and system based on a workflow task scheduling system.
Background
The Dolphin scheduler is a distributed and easily-extensible visual DAG workflow task scheduling system. Is a top level item of Apache. The Dolphin scheduler component can install a plurality of works, and when executing tasks, the tasks are distributed to different works for execution in a polling or load balancing mode according to the running state of the works host. And supporting the Worker grouping management, wherein the Worker is divided into different groups, and when the task is executed, the task can be run in the corresponding Worker grouping by selecting the corresponding Worker grouping.
In the framework of the existing dolphin scheduler, the existence check of the client is not supported when the task is run. When a task such as Flink, spark, MR, sqoop is executed, a host that is required to execute the task again installs a corresponding client. If the task is run, the client is found to be not installed, so that the task cannot run. Therefore, the client needs to be manually installed, which is very troublesome and cumbersome. And because the dolphin scheduler component is often installed together with the cluster component, MR clients are already installed on existing works, spark clients are installed on existing works, and it is very troublesome to comb the installation conditions of the clients. And because of the polling or load balancing mechanism of the workbench packet, tasks can be run successfully sometimes, and sometimes nodes which are not provided with corresponding clients can fail to run sometimes, so that the positioning is very troublesome in face of the problem of the occurrence.
Disclosure of Invention
Aiming at the problem that the existence check of a client is not supported when a task is run in the framework of the traditional Dolphin scheduler, the application provides a client management method, device and system based on a workflow task scheduling system.
The technical scheme of the application is as follows:
in a first aspect, the present application provides a method for managing a client based on a workflow task scheduling system, including the following steps:
judging whether a client is installed on a task execution end according to the environment variable of the system, and reporting and displaying a judging result;
when the client is not installed on the task execution end, the client is installed in a mode of downloading a package decompression configuration environment variable or in a yum source mode;
checking whether a client on a task execution end meets task operation or not; when the task operation is not satisfied, the task execution end grouping is replaced, or the client is installed for the task execution end which does not satisfy the task operation.
Further, in the step of judging whether the client has been installed on the task execution end according to the environment variable of the system and reporting and displaying the judgment result, the step of judging whether the client has been installed on the task execution end according to the environment variable of the system includes:
when a task execution end is started, initializing a scanning thread and configuring timing time;
and executing the command locally to acquire the installation condition of the client.
Further, the step of executing the command locally to obtain the installation condition of the client includes:
after receiving the input refreshing instruction, inserting a scanning client command into the database;
the database is scanned regularly, and after the inserted scanning client command is found, the command is packaged and issued to the task execution end;
and the task execution end executes the client scanning command according to the configured timing time to acquire the installation condition of the client.
Further, according to the environment variable of the system, judging whether the client is installed on the task execution end, and reporting the judging result for displaying, wherein the step of reporting the judging result for displaying comprises the following steps:
reporting the acquired installation condition of the client to a main management end;
packaging and warehousing the received information;
and displaying the information reported by scanning.
Further, when it is determined that the client is not installed on the task execution end, the step of installing the client by decompressing the configuration environment variable or the yum source by the download package includes:
when the client is not installed on the task execution end, receiving an input installation command;
selecting an installation mode;
inserting an installation command into the database when the client installation is performed by selecting a yum source mode;
when the scanning database discovers the client side installation command, the command is packaged and issued to the task execution side;
the task execution end orders the installation client through execution yum;
and reporting the installation result.
Further, when it is determined that the client is not installed on the task execution end, the step of installing the client by decompressing the configuration environment variable or the yum source by the download package further includes:
when the client installation is carried out by selecting a mode of decompressing and configuring environment variables by the download package, uploading the installation package to a distributed file system, and inserting an uploading installation command into a database;
when the scanning database finds that the uploading installation command is inserted, the command is packaged and issued to the task execution end;
the task execution end downloads a client installation package and decompresses configuration environment variables from the corresponding position of the distributed file system to finish the client installation; the method comprises the following steps: and reporting the installation result.
Further, checking whether the client on the task execution end meets the task operation; when the task operation is not satisfied, the step of replacing the task execution end group or installing the client for the task execution end which does not satisfy the task operation comprises the following steps:
receiving an input workflow execution command;
inquiring a database to check whether a task execution end group where the task operates meets a client condition;
if the condition is met, operating the workflow;
if the condition is not satisfied, outputting a prompt to judge whether to switch the task execution end group;
if a switch packet command is received, executing the steps: inquiring a database to check whether a task execution end group where the task operates meets a client condition;
if the switching grouping command is not received, outputting a prompt to judge whether the client is installed for the task execution end without the client;
if an installation command is received, executing the following steps: the method comprises the steps that client installation is carried out in a mode of downloading a package decompression configuration environment variable or in a yum source mode;
if the installation command is not received, outputting a prompt that the workflow cannot run, and ending.
In a second aspect, the present application further provides a client management device based on a workflow task scheduling system, which includes a client scanning reporting module, a client installation module and a workflow verification module;
the client scanning reporting module is used for judging whether the client is installed on the task execution end according to the environment variable of the system after the task execution end is installed, and reporting and displaying the judging result;
after the workbench is installed, judging whether the client is installed on the workbench according to the environment variable of the system, reporting to a Master at regular time, warehousing by the Master, displaying in a UI, and scanning and updating the installation mode of the client in a mode of supporting manual refreshing;
the client installation module is used for installing the client in a manner of decompressing the configuration environment variable of the download package or in a manner of yum source when the client is not installed on the task execution end;
the method can be used for carrying out one-key installation in a mode of decompressing configuration environment variables or a yum source mode through an HDFS download package;
the workflow verification module is used for verifying whether the client on the task execution end meets the task operation; and when the verification finds that the client does not meet the condition, replacing the task execution end group or installing the client for the task execution end which does not meet the condition.
Checking whether the task fails due to the fact that the client is not installed according to the situation of the Worker grouping where the Worker is operated; when the client is found to be unsatisfied, the client on the workbench can be ensured to meet the task operation by changing the workbench grouping, installing the client for unsatisfied workbench and the like, so that the smoothness of the task is improved.
In a third aspect, the present application provides a client management system based on a workflow task scheduling system, including an interaction module, a main management end, a task execution end and a client management module;
the task execution end is used for initializing a scanning thread when starting and configuring timing time; executing the received command, and reporting the command execution result to the main management end;
the main management end is used for packaging and warehousing the received information; the system is also used for regularly scanning the database, packaging the inserted command after finding the command, and issuing the command to the task execution end;
the interaction module is used for displaying the information reported by scanning; the system is also used for receiving input information and inserting corresponding commands into the database according to the received information;
the client management module is used for checking whether the client on the task execution end meets the task operation; when the verification finds that the client side does not meet the condition, the task execution end is replaced by grouping, or the task execution end which does not meet the condition outputs prompt information of the installation client side.
Further, the client management module is specifically configured to receive an input workflow execution command; inquiring a database to check whether a task execution end group where the task operates meets a client condition; if the condition is met, operating the workflow; if the condition is not satisfied, outputting a prompt to judge whether to switch the task execution end group; if the switching grouping command is not received, outputting a prompt to judge whether the client is installed for the task execution end without the client; if the installation command is not received, outputting a prompt that the workflow cannot run.
From the above technical scheme, the application has the following advantages: and the client-side installation condition on a workbench on the Dolphin scheduler component is managed by client-side scanning and reporting, client-side installation and workflow execution verification. When the method is used, the installation condition, the workbench state and the grouping of the workbench of the client on the workbench are clear at a glance; in the aspect of client installation, the client can be installed in a one-key way by means of decompressing and configuring environment variables by the HDFS download package and yum sources, so that the client is simple and easy to use; in the aspect of workflow execution, before execution, the workflow is checked according to the workbench condition in the workbench group, so that the task is prevented from failing due to the fact that the client is not installed.
In addition, the application has reliable design principle, simple structure and very wide application prospect.
It can be seen that the present application has outstanding substantial features and significant advances over the prior art, as well as its practical advantages.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required to be used in the description of the embodiments or the prior art will be briefly described below, and it will be obvious to those skilled in the art that other drawings can be obtained from these drawings without inventive effort.
FIG. 1 is a schematic flow chart of a method of one embodiment of the application.
Fig. 2 is a schematic diagram of a scanning reporting procedure in a method according to an embodiment of the present application.
FIG. 3 is a schematic diagram of an installation flow of a method of one embodiment of the application.
FIG. 4 is a schematic diagram of a verification flow of a method according to an embodiment of the application.
Detailed Description
The Dolphin scheduler is a distributed and easily-extensible visual DAG workflow task scheduling system. The method is a top-level project of Apache, and the Dolphin scheduler is the same as a workflow task scheduling system in the application;
the dolphin scheduler architecture includes the following:
UI and API: and the interaction module is used for carrying out configuration management DAG, managing file resources required by task operation and monitoring data sources.
Master cluster: and resolving and splitting the DAG, and distributing tasks in the DAG to a workbench, wherein a Master is the same as a Master management end in the application.
Worker Cluster: and downloading resource files required by executing the tasks from the file system, executing the specific tasks, monitoring the states of the tasks and reporting the states to the Master. Worker is the same as the task execution end in the application.
Tasks: supporting running Flink, MR, spark and other tasks in the cluster. Through the combination of the modules, the dolphin schedule has two main functions:
1. various tasks such as monitoring management Flink, MR, spark and the like are scheduled at regular time according to the sequence of the DAG, so that the operation and maintenance cost of big data development is reduced.
2. The DAG graph and task information in the DAG graph are configured in a visual dragging mode, so that the method is convenient and easy to use.
The Dolphin scheduler component can install a plurality of works, and when executing tasks, the tasks are distributed to different works for execution in a polling or load balancing mode according to the running state of the works host. And supporting the Worker grouping management, wherein the Worker is divided into different groups, and when the task is executed, the task can be run in the corresponding Worker grouping by selecting the corresponding Worker grouping.
The Dolphin scheduler is provided with a file management function, resources Jar package and files needed by the task can be uploaded through the function, after the files are uploaded, the Dolphin scheduler stores the files into a configured file system (such as HDFS, S3 and the like), and when the task is executed, the worker can download the files stored in the file system according to the information of the task configuration, so that the task is executed.
However, in the framework of the existing dolphin scheduler, the presence check of the client is not supported when running tasks. When a task such as Flink, spark, MR, sqoop is executed, a host that is required to execute the task again installs a corresponding client. If the task is run, the client is found to be not installed, so that the task cannot run. Therefore, the client needs to be manually installed, which is very troublesome and cumbersome. And because the dolphin scheduler component is often installed together with the cluster component, MR clients are already installed on existing works, spark clients are installed on existing works, and it is very troublesome to comb the installation conditions of the clients. And because of the polling or load balancing mechanism of the workbench packet, tasks can be run successfully sometimes, and sometimes nodes which are not provided with corresponding clients can fail to run sometimes, so that the positioning is very troublesome in face of the problem of the occurrence.
The application designs a client management mode based on a Dolphin scheduler, which is divided into three processes of client scanning and reporting, client installation and workflow execution and verification, and manages the client installation condition on a workbench on the Dolphin scheduler component. In the process of client scanning and reporting, after the workbench is installed, the workbench judges whether the client is installed on the workbench according to the environment variable of the system, the workbench is reported to a Master at regular time, the Master is used for warehousing, the UI is used for displaying, and the installation condition of the client on the workbench is clear at a glance when the workbench is used; in the installation of the client, the client can be installed in a one-key way by means of decompressing and configuring environment variables by the HDFS download package and yum sources, so that the client is simple and easy to use; in the workflow execution checking module, whether the workflow fails due to the fact that the client is not installed or not is checked according to the condition of the Worker group where the running Worker is located.
In order to make the technical solution of the present application better understood by those skilled in the art, the technical solution of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, shall fall within the scope of the present application. Note that MySQL: traditional relational databases.
HDFS: a distributed file system.
MapReduce: mapReduce is a programming model for parallel operation of large-scale datasets (greater than 1 TB).
Flink: apache Flink is an open source stream processing framework developed by the Apache software Foundation.
As shown in fig. 1, an embodiment of the present application provides a client management method based on a workflow task scheduling system, including the following steps:
step 1: after the task execution end is installed, judging whether a client is installed on the task execution end according to the environment variable of the system, and reporting and displaying the judging result;
after the workbench is installed by the Dolphin scheduler, the workbench can judge whether a client is installed on the workbench according to environment variables of the system, report the client to a Master at regular time, store the client in the Master, display the client in a UI, and trigger a scanning flow manually at the UI interface.
The IP of the workbench and the workbench grouping can be displayed on the interface, and the operation state of the workbench and the installed client side. Information about the Worker shown relative to the Worker grouping angle, which section shows information about the Worker from the Worker's perspective.
In the operation, two buttons, namely a refreshing button and an installation button, are adopted, and after clicking the refreshing button, the workbench actively scans the installation condition of the client and updates the state. The client state of the workbench can be refreshed in batches by checking the check boxes;
step 2: when the client is not installed on the task execution end, the client is installed in a mode of downloading a package decompression configuration environment variable or in a yum source mode; note that the yum source is software installation through the yum source.
Step 3: checking whether a client on a task execution end meets task operation or not; and when the verification finds that the client does not meet the condition, replacing the task execution end group or installing the client for the task execution end which does not meet the condition.
In some embodiments, after the task execution end is installed in step 1, according to the environment variable of the system, judging whether the client is installed on the task execution end, and reporting and displaying the judging result, according to the environment variable of the system, the step of judging whether the client is installed on the task execution end includes:
step 11: when a task execution end is started, initializing a scanning thread and configuring timing time;
initializing a scanning thread and configuring timing time when a workbench is started;
step 12: executing the command locally to obtain the installation condition of the client; the workbench locally executes the command, acquires the installation condition of the client, reports the Master cluster, does not directly put in storage or report the API, reduces the pressure of a database, and simultaneously avoids the failure of the API to report information;
it should be noted that the steps specifically include:
step 121: after receiving the input refreshing instruction, inserting a scanning client command into the database;
after the interface clicks and refreshes, a scanning client command is inserted into the database;
step 122: the database is scanned regularly, and after the inserted scanning client command is found, the command is packaged and issued to the task execution end; the Master scans the database regularly, packages the command after finding the command, and issues the command to the workbench;
step 123: and the task execution end executes the client scanning command according to the configured timing time to acquire the installation condition of the client. And the workbench executes the client scanning according to the timing scanning flow.
Step 13: reporting the acquired installation condition of the client to a main management end;
step 14: packaging and warehousing the received information; packaging and warehousing information by a Master;
step 15: and displaying the information reported by scanning. And the API & UI module displays the information reported by scanning. In the application, the API & UI module is the same as the interaction module described below.
In some embodiments, in step 2, when it is determined that the client is not installed on the task execution end, the step of performing client installation by means of decompressing the configuration environment variable or by means of yum source by the download package includes:
step 21: when the client is not installed on the task execution end, receiving an input installation command; inputting an installation command on the interface by clicking an installation button;
step 22: selecting an installation mode; step 23a is executed when the client installation is performed by selecting a yum source mode, and step 23b is executed when the client installation is performed by selecting a download package decompression configuration environment variable mode;
step 23a: inserting an installation command into the database; the API & UI module inserts an installation command in the database, wherein the installation command comprises information such as IP of an installation workbench, installation type and the like;
step 24a: when the scanning database discovers the client side installation command, the command is packaged and issued to the task execution side; the execution main body of the step is a Master, a Master scanning database discovers a client scanning command, packages and transmits the client scanning command to a workbench;
step 25a: the task execution end orders the installation client through execution yum;
step 26: and reporting the installation result.
Step 23b: uploading the installation package to a distributed file system, and inserting an uploading installation command into a database; the API & UI module uploads an installation package to the HDFS through an interface by the existing file management module, and meanwhile, a database inserts a manual uploading installation command, wherein the installation command comprises information such as IP of an installation workbench, an installation type, the position of the HDFS stored in the installation package and the like;
step 24b: when the scanning database finds that the uploading installation command is inserted, the command is packaged and issued to the task execution end;
step 25b: the task execution end downloads a client installation package and decompresses configuration environment variables from the corresponding position of the distributed file system to finish the client installation; step 26 is performed;
in some embodiments, in step 3, checking whether the client on the task execution end satisfies the task operation; when the verification finds that the client does not meet the condition, the step of replacing the task execution end group or installing the client for the task execution end which does not meet the condition comprises the following steps:
step 31: receiving an input workflow execution command; clicking and executing the workflow on the interface;
step 32: inquiring a database to check whether a task execution end group where the task operates meets a client condition; if the condition is satisfied, step 35 is performed; if the condition is not satisfied, step 33 is executed;
step 33: outputting a prompt to judge whether to switch the task execution end group; if a handover packet command is received, step 32 is performed; if the handover packet command is not received, step 34 is performed;
step 34: outputting a prompt to judge whether to install the client for the task execution end without installing the client; if an installation command is received, step 22 is executed; if no installation command is received, step 36 is performed;
step 35: operating a workflow;
step 36: and outputting a prompt that the workflow cannot run and ending.
The embodiment of the application also provides a client management device based on the workflow task scheduling system, which comprises a client scanning and reporting module, a client installation module and a workflow verification module;
the client scanning reporting module is used for judging whether the client is installed on the task execution end according to the environment variable of the system after the task execution end is installed, and reporting and displaying the judging result;
after the workbench is installed by the Dolphin scheduler, the workbench can judge whether a client is installed on the workbench according to environment variables of the system, report the client to a Master at regular time, store the client in the Master, display the client in a UI, and trigger a scanning flow manually at the UI interface.
The IP of the workbench and the workbench grouping can be displayed on the interface, and the operation state of the workbench and the installed client side. Information about the Worker shown relative to the Worker grouping angle, which section shows information about the Worker from the Worker's perspective.
In the operation, two buttons, namely a refreshing button and an installation button, are adopted, and after clicking the refreshing button, the workbench actively scans the installation condition of the client and updates the state. The client state of the workbench can be refreshed in batches by checking the check boxes;
as shown in fig. 2, the specific flow of the workbench scanning flow and the timing reporting flow are as follows:
initializing a scanning thread and configuring timing time when a workbench is started;
the workbench locally executes the command, acquires the installation condition of the client, reports the Master cluster, does not directly put in storage or report the API, reduces the pressure of the database, and simultaneously avoids the failure of the API to report information;
packaging and warehousing the information by using a Master;
an API & UI module displays the information reported by scanning;
the API issuing command flow is as follows:
1. after the interface clicks and refreshes, a scanning client command is inserted into the database;
scanning the database at regular time by a Master, packaging the command after finding the command, and issuing the command to a workbench;
the workbench executes client scanning according to the timing scanning flow;
the client installation module is used for installing the client in a manner of decompressing the configuration environment variable of the download package or in a manner of yum source when the client is not installed on the task execution end;
the one-key installation can be performed by means of decompressing the configuration environment variable by the HDFS download package or by means of yum source, as shown in fig. 3, the specific flow is as follows:
1. clicking an installation command on an interface, selecting an installation mode, if the installation of the Yum source is selected to be executed 2.1, and if the installation of the Jar package is uploaded to be executed 3.1;
2.1api & ui: the database is inserted with an installation command, and the installation command comprises information such as IP of an installation workbench, installation type and the like;
master 2.2: the scanning database discovers a client scanning command, encapsulates the client scanning command and sends the client scanning command to a workbench;
2.3 workbench: installing the client by executing the YUM command, and executing the step 4 after the installation is completed;
3.1api & ui: uploading an installation package to an HDFS through an interface by an existing file management module, and inserting a manual uploading installation command into a database, wherein the installation command comprises information such as IP of an installation workbench, installation type, the position of the HDFS stored in the installation package and the like;
3.2Master: the scanning database discovers a client scanning command, encapsulates the client scanning command and sends the client scanning command to a workbench;
3.3 workbench: downloading a client installation package and decompressing configuration environment variables from the corresponding position of the HDFS to finish installation;
workbench: reporting the installation result;
5. and exposing the installation result at the API and the UI module.
The workflow verification module is used for verifying whether the client on the task execution end meets the task operation; and when the verification finds that the client does not meet the condition, replacing the task execution end group or installing the client for the task execution end which does not meet the condition.
In the workflow execution checking module, according to the condition of the Worker group where the running Worker is located, whether the workflow fails due to the fact that the client is not installed is checked, as shown in fig. 4, the specific execution flow is as follows:
1. clicking and executing the workflow on the interface;
2. inquiring a database to check whether the workbench group where the task operates meets the client condition or not, and not meeting the execution 3; execution 6 is satisfied;
3. prompting whether to switch the workbench group, if yes, re-executing 2, otherwise executing 4;
4. prompting whether to install the client for the workbench without installing the client, if so, executing 5, otherwise, executing 7;
5. installing a client for a workbench without installing the client according to the client flow;
6. operating a workflow;
7. prompting that the workflow cannot run and ending;
the embodiment of the application also provides a client management system based on the workflow task scheduling system, which comprises an interaction module, a main management end, a task execution end and a client management module;
the task execution end is used for initializing a scanning thread when starting and configuring timing time; executing the received command, and reporting the command execution result to the main management end;
the main management end is used for packaging and warehousing the received information; the system is also used for regularly scanning the database, packaging the inserted command after finding the command, and issuing the command to the task execution end;
the interaction module is used for displaying the information reported by scanning; the system is also used for receiving input information and inserting corresponding commands into the database according to the received information;
the client management module is used for checking whether the client on the task execution end meets the task operation; when the verification finds that the client side does not meet the condition, the task execution end is replaced by grouping, or the task execution end which does not meet the condition outputs prompt information of the installation client side.
The client management module is specifically used for receiving an input workflow execution command; inquiring a database to check whether a task execution end group where the task operates meets a client condition; if the condition is met, operating the workflow; if the condition is not satisfied, outputting a prompt to judge whether to switch the task execution end group; if the switching grouping command is not received, outputting a prompt to judge whether the client is installed for the task execution end without the client; if the installation command is not received, outputting a prompt that the workflow cannot run.
Although the present application has been described in detail by way of preferred embodiments with reference to the accompanying drawings, the present application is not limited thereto. Various equivalent modifications and substitutions may be made in the embodiments of the present application by those skilled in the art without departing from the spirit and scope of the present application, and it is intended that all such modifications and substitutions be within the scope of the present application/be within the scope of the present application as defined by the appended claims. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (10)
1. A client management method based on a workflow task scheduling system is characterized by comprising the following steps:
judging whether a client is installed on a task execution end according to the environment variable of the system, and reporting and displaying a judging result;
when the client is not installed on the task execution end, the client is installed in a mode of downloading a package decompression configuration environment variable or in a yum source mode;
checking whether a client on a task execution end meets task operation or not; when the task operation is not satisfied, the task execution end grouping is replaced, or the client is installed for the task execution end which does not satisfy the task operation.
2. The method for managing clients of a workflow task scheduling system according to claim 1, wherein in the step of judging whether a client is installed on a task execution terminal according to an environment variable of the system and reporting and displaying the judgment result, the step of judging whether the client is installed on the task execution terminal according to the environment variable of the system comprises:
when a task execution end is started, initializing a scanning thread and configuring timing time;
and executing the command locally to acquire the installation condition of the client.
3. The method for managing clients based on the workflow task scheduling system of claim 2, wherein the step of obtaining the installation condition of the clients by locally executing the command comprises:
after receiving the input refreshing instruction, inserting a scanning client command into the database;
the database is scanned regularly, and after the inserted scanning client command is found, the command is packaged and issued to the task execution end;
and the task execution end executes the client scanning command according to the configured timing time to acquire the installation condition of the client.
4. The method for managing clients based on a workflow task scheduling system according to claim 2, wherein in the step of judging whether a client is installed on a task execution end according to an environment variable of the system and reporting and displaying the judgment result, the step of reporting and displaying the judgment result includes:
reporting the acquired installation condition of the client to a main management end;
packaging and warehousing the received information;
and displaying the information reported by scanning.
5. The method for managing clients in a workflow task scheduling system according to claim 2, wherein when it is determined that a client is not installed on a task execution side, the step of installing the client by decompressing a configuration environment variable by a download package or by a yum source includes:
when the client is not installed on the task execution end, receiving an input installation command;
selecting an installation mode;
inserting an installation command into the database when the client installation is performed by selecting a yum source mode;
when the scanning database discovers the client side installation command, the command is packaged and issued to the task execution side;
the task execution end orders the installation client through execution yum;
and reporting the installation result.
6. The method for managing clients in a workflow task scheduling system of claim 5, wherein when it is determined that a client is not installed on a task execution side, the step of installing the client by decompressing a configuration environment variable by a download package or by a yum source further comprises:
when the client installation is carried out by selecting a mode of decompressing and configuring environment variables by the download package, uploading the installation package to a distributed file system, and inserting an uploading installation command into a database;
when the scanning database finds that the uploading installation command is inserted, the command is packaged and issued to the task execution end;
the task execution end downloads a client installation package and decompresses configuration environment variables from the corresponding position of the distributed file system to finish the client installation; the method comprises the following steps: and reporting the installation result.
7. The method for managing clients based on a workflow task scheduling system according to claim 2, wherein it is checked whether the clients on the task execution side satisfy task operations; when the task operation is not satisfied, the step of replacing the task execution end group or installing the client for the task execution end which does not satisfy the task operation comprises the following steps:
receiving an input workflow execution command;
inquiring a database to check whether a task execution end group where the task operates meets a client condition;
if the condition is met, operating the workflow;
if the condition is not satisfied, outputting a prompt to judge whether to switch the task execution end group;
if a switch packet command is received, executing the steps: inquiring a database to check whether a task execution end group where the task operates meets a client condition;
if the switching grouping command is not received, outputting a prompt to judge whether the client is installed for the task execution end without the client;
if an installation command is received, executing the following steps: client installation is carried out by means of decompression configuration environment variables of a download package or yum sources;
if the installation command is not received, outputting a prompt that the workflow cannot run, and ending.
8. The client management device based on the workflow task scheduling system is characterized by comprising a client scanning reporting module, a client installing module and a workflow checking module;
the client scanning reporting module is used for judging whether the client is installed on the task execution end according to the environment variable of the system after the task execution end is installed, and reporting and displaying the judging result;
the client installation module is used for installing the client in a manner of decompressing the configuration environment variable of the download package or in a manner of yum source when the client is not installed on the task execution end;
the workflow verification module is used for verifying whether the client on the task execution end meets the task operation; and when the verification finds that the client does not meet the condition, replacing the task execution end group or installing the client for the task execution end which does not meet the condition.
9. The client management system based on the workflow task scheduling system is characterized by comprising an interaction module, a main management end, a task execution end and a client management module; the system further comprising the apparatus of claim 8;
the task execution end is used for initializing a scanning thread when starting and configuring timing time; executing the received command, and reporting the command execution result to the main management end;
the main management end is used for packaging and warehousing the received information; the system is also used for regularly scanning the database, packaging the inserted command after finding the command, and issuing the command to the task execution end;
the interaction module is used for displaying the information reported by scanning; the system is also used for receiving input information and inserting corresponding commands into the database according to the received information;
the client management module is used for checking whether the client on the task execution end meets the task operation; when the verification finds that the client side does not meet the condition, the task execution end is replaced by grouping, or the task execution end which does not meet the condition outputs prompt information of the installation client side.
10. The client management system based on a workflow task scheduling system of claim 9, wherein the client management module is specifically configured to receive an input workflow execution command; inquiring a database to check whether a task execution end group where the task operates meets a client condition; if the condition is met, operating the workflow; if the condition is not satisfied, outputting a prompt to judge whether to switch the task execution end group; if the switching grouping command is not received, outputting a prompt to judge whether the client is installed for the task execution end without the client; if the installation command is not received, outputting a prompt that the workflow cannot run.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210044858.2A CN114443292B (en) | 2022-01-14 | 2022-01-14 | Client management method, device and system based on workflow task scheduling system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210044858.2A CN114443292B (en) | 2022-01-14 | 2022-01-14 | Client management method, device and system based on workflow task scheduling system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114443292A CN114443292A (en) | 2022-05-06 |
| CN114443292B true CN114443292B (en) | 2023-11-14 |
Family
ID=81368609
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210044858.2A Active CN114443292B (en) | 2022-01-14 | 2022-01-14 | Client management method, device and system based on workflow task scheduling system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114443292B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113254010A (en) * | 2021-07-09 | 2021-08-13 | 广州光点信息科技有限公司 | Visual DAG workflow task scheduling system and operation method thereof |
| CN113672452A (en) * | 2021-07-23 | 2021-11-19 | 济南浪潮数据技术有限公司 | Method and system for monitoring operation of data acquisition task |
| CN113687932A (en) * | 2021-08-30 | 2021-11-23 | 上海商汤科技开发有限公司 | Task scheduling method, device and system, electronic equipment and storage medium |
| CN113687927A (en) * | 2021-07-31 | 2021-11-23 | 济南浪潮数据技术有限公司 | Method, device, equipment and storage medium for scheduling and configuring flash tasks |
-
2022
- 2022-01-14 CN CN202210044858.2A patent/CN114443292B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113254010A (en) * | 2021-07-09 | 2021-08-13 | 广州光点信息科技有限公司 | Visual DAG workflow task scheduling system and operation method thereof |
| CN113672452A (en) * | 2021-07-23 | 2021-11-19 | 济南浪潮数据技术有限公司 | Method and system for monitoring operation of data acquisition task |
| CN113687927A (en) * | 2021-07-31 | 2021-11-23 | 济南浪潮数据技术有限公司 | Method, device, equipment and storage medium for scheduling and configuring flash tasks |
| CN113687932A (en) * | 2021-08-30 | 2021-11-23 | 上海商汤科技开发有限公司 | Task scheduling method, device and system, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114443292A (en) | 2022-05-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3769223B1 (en) | Unified test automation system | |
| US20210342313A1 (en) | Autobuild log anomaly detection methods and systems | |
| US20140282421A1 (en) | Distributed software validation | |
| US8037453B1 (en) | System and method for continuous software configuration, test and build management | |
| CN109885316B (en) | Hdfs-hbase deployment method and device based on kubernetes | |
| US8762929B2 (en) | System and method for exclusion of inconsistent objects from lifecycle management processes | |
| US9038059B2 (en) | Automatically targeting application modules to individual machines and application framework runtimes instances | |
| CN106293863B (en) | The method and system of system function dynamic expansion replacement are realized in multilingual cloud compiling | |
| US10466981B1 (en) | System and method for generative programming in an integrated development environment (IDE) | |
| CN113434158B (en) | Custom management method, device, equipment and medium for big data component | |
| CN114327678B (en) | Real-time data processing system and method supporting multiple engines | |
| CN111666189A (en) | Method and system for declaratively visually configuring Prometheus monitoring alarm | |
| US20200133711A1 (en) | Event-Triggered Configuration of Workflow Processes for Computer Software Development Systems | |
| CN109240716B (en) | Big data platform version management and rapid iterative deployment method and system | |
| US20130275960A1 (en) | Validation of current states of provisioned software products in a cloud environment | |
| CN110704082A (en) | Continuous integration method and system for monitoring platform | |
| CN112860282A (en) | Upgrading method and device of cluster plug-in and server | |
| CN111831567A (en) | Application test environment configuration method, device, system and medium | |
| CN115220863A (en) | Operation and maintenance method and device for container application, computer equipment and storage medium | |
| US9760364B2 (en) | Checks for software extensions | |
| CN113961570A (en) | Real-time acquisition method applied to MYSQL BINLog change data | |
| CN114443292B (en) | Client management method, device and system based on workflow task scheduling system | |
| CN105279065A (en) | Method and apparatus for making statistics on test results in cloud test platform | |
| US8805955B2 (en) | Proactive caching of remote actions | |
| US11003575B1 (en) | Systems and methods for continuous integration automated testing in a distributed computing environment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |