CN108710532B - Dependency realization method, device, equipment and storage medium of cross-dispatching platform - Google Patents
Dependency realization method, device, equipment and storage medium of cross-dispatching platform Download PDFInfo
- Publication number
- CN108710532B CN108710532B CN201810486735.8A CN201810486735A CN108710532B CN 108710532 B CN108710532 B CN 108710532B CN 201810486735 A CN201810486735 A CN 201810486735A CN 108710532 B CN108710532 B CN 108710532B
- Authority
- CN
- China
- Prior art keywords
- scheduling
- platform
- scheduling task
- task
- task identifier
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention discloses a method, a device, equipment and a storage medium for realizing dependence of a cross-dispatching platform, wherein the method comprises the following steps: the first scheduling platform acquires a first scheduling task identifier; the first scheduling platform searches a second scheduling task identifier corresponding to the first scheduling task identifier in a first configuration table; the first scheduling platform polls whether a log table of a second scheduling platform has a mark file which is corresponding to the second scheduling task identifier and is successful in running or not by using a cross-platform scripting language; and if the log table of the second scheduling platform contains a mark file which is used for successfully operating the second scheduling task and corresponds to the second scheduling task identifier, the first scheduling platform executes the first scheduling task corresponding to the first scheduling task identifier. The method is convenient for searching and monitoring the operation result of the dispatching task, and realizes the dependence of the task between dispatching platforms.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method, an apparatus, a device, and a storage medium for implementing cross-scheduling platform dependency.
Background
At present, big data tasks are various in operation types and operate on different dispatching platforms, and because the operation modes and rules of the tasks of the different dispatching platforms are different, data cannot be directly relied on, so that effective and strict data flow cannot be formed, and a developer can only realize the dependence by timing, so that the problems of incomplete data, overlong waiting time and the like are easy to occur.
For example, there is an item a, where an OOZIE (OOZIE is a server based on a workflow engine) is used to schedule tasks, and data exists in a Hadoop HIVE (Hadoop, hadoop Distributed File System, HDFS for short, which is a distributed system infrastructure, HIVE is a data warehouse architecture built on a Hadoop file system, and analyzes and manages data stored in the HDFS) table; in addition, an item B adopts linkdo (linkdo is a distributed task scheduling platform) to schedule hadoop tasks, and the data of the item B tasks depend on the generation of the data in the HIVE table in the item A. If no dependency is formed, the execution can be realized only by staggering the execution time, for example, the task of the A project runs at 5 points, and the task of the B project can be executed only after 8 points if the execution is expected to be completed in 1-3 hours.
Because the running time of the task fluctuates every day, if the task of the project A is completed in 1 hour, the project B waits for two more hours because the project B is set to be executed at 8 points; or the task of item B is executed, the data of the task of item a has not yet been completely generated.
Disclosure of Invention
Based on the above, it is necessary to provide a method, a device, equipment and a storage medium for implementing cross-scheduling platform dependence, aiming at the problems of long task running time and data error in the prior art.
A dependency implementation method across scheduling platforms, the dependency implementation method comprising:
the first scheduling platform acquires a first scheduling task identifier;
the first scheduling platform searches a second scheduling task identifier corresponding to the first scheduling task identifier in a first configuration table, wherein the execution of the first scheduling task corresponding to the first scheduling task identifier depends on the execution of the second scheduling task corresponding to the second scheduling task identifier;
the first scheduling platform polls whether a log table of a second scheduling platform has a mark file which is corresponding to the second scheduling task identifier and is successful in running or not by using a cross-platform scripting language;
and if the log table of the second scheduling platform contains a mark file which is used for successfully operating the second scheduling task and corresponds to the second scheduling task identifier, the first scheduling platform executes the first scheduling task corresponding to the first scheduling task identifier.
In one embodiment, the method further comprises:
and if the log table of the second scheduling platform does not contain a mark file which is corresponding to the second scheduling task identifier and is successful in running the second scheduling task, the first scheduling platform calls a monitoring program of the second scheduling platform to check the execution progress of the second scheduling task.
In one embodiment, the first scheduling platform and the second scheduling platform are any one of Oozie, linkdo, control-M, XXL-JOB or tivoliworkload scheduler.
In one embodiment, the cross-platform scripting language is any one of Python, perl, ruby, PHP, lua or Java scripting language.
A dependency realization apparatus across a dispatch platform, the realization apparatus comprising:
the acquisition module is used for acquiring a first scheduling task identifier by the first scheduling platform;
the searching module is used for searching a second scheduling task identifier corresponding to the first scheduling task identifier in a first configuration table by the first scheduling platform, wherein the execution of the first scheduling task corresponding to the first scheduling task identifier depends on the execution of the second scheduling task corresponding to the second scheduling task identifier;
the polling module is used for polling whether a log table of the second scheduling platform has a mark file which is corresponding to the second scheduling task identifier and is successful in running or not by using a cross-platform scripting language by the first scheduling platform;
and the execution module is used for executing the first scheduling task corresponding to the first scheduling task identifier by the first scheduling platform if the mark file which is used for successfully running the second scheduling task corresponding to the second scheduling task identifier exists in the log table of the second scheduling platform.
In one embodiment, the cross-platform dependency implementation apparatus further includes a monitoring module, configured to, if a log table of the second scheduling platform does not have a flag file that the second scheduling task corresponding to the second scheduling task identifier runs successfully, call a monitor program of the second scheduling platform to check execution progress of the second scheduling task.
In one embodiment, the first scheduling platform and the second scheduling platform are any one of Oozie, linkdo, control-M, XXL-JOB or tivoliworkload scheduler.
In one embodiment, the cross-platform scripting language is any one of Python, perl, ruby, PHP, lua or Java scripting language.
A computer device comprising a memory and a processor, the memory having stored therein computer readable instructions which, when executed by the processor, cause the processor to perform the steps of the above-described implementation method.
A storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the above-described implementation method.
According to the cross-scheduling platform dependency realization method, device, equipment and storage medium, the first scheduling task identification is obtained through the first scheduling platform, the first scheduling platform searches the second scheduling task identification corresponding to the first scheduling task identification in the first configuration table, wherein the execution of the first scheduling task corresponding to the first scheduling task identification depends on the execution of the second scheduling task corresponding to the second scheduling task identification, the first scheduling platform polls whether the log table of the second scheduling platform has a mark file which is successful in running of the second scheduling task corresponding to the second scheduling task identification by using a cross-platform scripting language, if the log table of the second scheduling platform has a mark file which is successful in running of the second scheduling task corresponding to the second scheduling task identification, the first scheduling platform executes the first scheduling task corresponding to the first scheduling task identification, and if the log table of the second scheduling platform does not have a mark file which is successful in running of the second scheduling task corresponding to the second scheduling task identification, the first scheduling platform calls the second scheduling task to execute the mark file, so that the dependency of the dependency platform is not completely run through, and the realization of the dependency platform dependency realization is realized.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention.
FIG. 1 is a flow diagram of a method of dependency implementation across a dispatch platform in one embodiment;
FIG. 2 is a block diagram of a dependency implementation across a dispatch platform in one embodiment;
FIG. 3 is a block diagram of the internal architecture of a computer device in one embodiment.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless expressly stated otherwise, as understood by those skilled in the art. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
As a preferred embodiment, as shown in fig. 1, a dependency implementation method across a scheduling platform, the dependency implementation method includes:
step S101, a first scheduling platform acquires a first scheduling task identifier; the first scheduling platform is a distributed scheduling platform and can be any one of platforms Oozie, linkdo, control-M, XXL-JOB, tivoliWorkloadScheduler; wherein, the liquid crystal display device comprises a liquid crystal display device,
the distributed task scheduling platform XXL-JOB is an open source framework, is based on an open source Quartz scheduling kernel, is an integral scheduling task, cannot edit a large task flow chart, and tracks the execution state of each node. Dependency relationships cannot be configured among multiple tasks and the tasks run sequentially, for example, an upstream task signals a downstream task. IBM Tivoli Workload Scheduler may correspond to cron under Unix, but it enhances many functions for enterprise scheduling, such as scheduling can be handled based on dependencies and event drivers, multi-time zone management, etc. cron can only schedule on a single server based on time, while TWS can replace cron to process job scheduling with its own daemon with richer functionality. Oozie is a workflow engine server for running workflow of tasks such as hadoop map/reduce and hive. Control-M adopts C/S mode, installs Enterprise Manager and server on server, installs agent on controlled host, agent can submit job flow defined by Control-M on host, and returns operation result. The JOB's operating conditions can be monitored throughout all batches, and JOB's operating conditions and process interventions can be controlled in a number of ways, as per Enterprise Manager.
The first scheduling task identifier is used for uniquely distinguishing a scheduling task to be executed on the first scheduling platform, and may be a combination string of letters, numbers and underlines of a preset rule. In this embodiment, the specific implementation steps of the first scheduling platform to obtain the first scheduling task may be: the first scheduling platform sequentially scans the current state of each task in a preset scheduling task list, and if the current state is a ready state or a ready state, acquires a scheduling task identifier corresponding to the ready state or the ready state. The scheduling task list is provided with scheduling task information such as scheduling task names, scheduling task identifiers, scheduling task states and the like; the scheduled task state is used to characterize the current state of the scheduled task, including the ready state and the ready state. It will be appreciated that in this embodiment, the scheduling task information is stored separately.
In other embodiments, the scheduled task information may be stored in the same task table with non-scheduled task information of the first scheduled platform (i.e., task information that the first scheduled platform performs itself without performing by calling other platforms), and then the specific implementation steps of the first scheduled platform to obtain the first scheduled task may be: the first scheduling platform sequentially reads the task type of each task from a preset task list, if the task type is a scheduling task, the current state of the scheduling task is obtained, and if the current state is a ready state or a ready state, the scheduling task identification corresponding to the ready state or the ready state is obtained.
Step S102, the first scheduling platform searches a second scheduling task identifier corresponding to the first scheduling task identifier in a first configuration table, wherein the execution of the first scheduling task corresponding to the first scheduling task identifier depends on the execution of the second scheduling task corresponding to the second scheduling task identifier;
the second dispatch platform is a dispatched platform corresponding to the first dispatch platform and adopting a different frame from the first dispatch platform, for example, the first dispatch platform adopts a XXL-JOB frame platform, and then the second dispatch platform can be a platform adopting any frame of Oozie, linkdo, control-M or Tivoli workbench scheduler; the second scheduling platform identifier is used for uniquely distinguishing scheduling tasks to be executed on the second scheduling platform, and can be a combination string of letters, numbers and underlines of a preset rule; the first configuration table is arranged on the first scheduling platform and stores a mapping relation between task identifiers of the first scheduling platform and task identifiers of the second scheduling platform, wherein the execution of the first scheduling task corresponding to the first scheduling task identifier depends on the execution of the second scheduling task corresponding to the second scheduling task identifier, namely, after the second scheduling task corresponding to the second scheduling platform identifier is successfully executed, the first scheduling task corresponding to the first scheduling platform identifier is executed.
The following table is an example of field setting of a first configuration table in an embodiment, in this embodiment, a first scheduling platform is a platform of an XXL-JOB framework, and a second scheduling platform is a platform of an Oozie framework, where a field oozie_task_name is a first scheduling task identifier, and a field task_name is a second scheduling task identifier:
TABLE 1
Step S103, the first scheduling platform polls whether a log table of a second scheduling platform has a mark file which is corresponding to the second scheduling task identifier and is successful in running or not by using a cross-platform scripting language;
the log table is preset on the second scheduling platform and is used for receiving a mark file of successful operation of the second scheduling task. After the second scheduling platform executes the second scheduling task, a mark file is generated according to the running condition of the scheduling task, and the mark file is stored in the created log table.
The following table is an example of a log table in one embodiment, where RUN-DATA is the running DATA, OOZIE-TASK-NAME is the OOZIE TASK NAME, TASK-NAME is the TASK NAME, and CHECK-TIME is the CHECK TIME.
TABLE 2
In this embodiment, a cross-platform scripting language is called by the crontab of LINUX, one of Python, perl, ruby, PHP, lua and Java may be selected as the cross-platform scripting language, the selected cross-platform scripting language is triggered once every one minute by the crontab function of LINUX, and the cross-platform scripting language uses the cross-platform scripting language at the first scheduling platform every one minute by using the detection function thereof to poll whether the log table of the second scheduling platform has a mark file that the second scheduling task corresponding to the second scheduling task identifier runs successfully, and if the log table of the second scheduling platform has a mark file that the second scheduling task corresponding to the second scheduling task identifier runs successfully, the virtual task in the scheduling platform is set to a completion state.
Step S104, if a mark file which is successful in running of the second scheduling task and corresponds to the second scheduling task identifier exists in the log table of the second scheduling platform, the first scheduling platform executes the first scheduling task corresponding to the first scheduling task identifier.
In this embodiment, if a flag file that the second scheduling task corresponding to the second scheduling task identifier runs successfully exists in the log table of the second scheduling platform, the first scheduling platform executes the first scheduling task corresponding to the first scheduling task identifier. The first scheduling platform is an active party and the second scheduling platform is a passive party. The first scheduling platform initiates a query, queries the mark in the second scheduling platform, and if the mark is completed, the first scheduling platform triggers the task of the first scheduling platform to start running so as to achieve the aim of dependence. The flag may be agreed, and may be a file with a time and task tag or a field in a database.
In one embodiment, if the log table of the second scheduling platform does not have a flag file that the second scheduling task corresponding to the second scheduling task identifier runs successfully, the first scheduling platform invokes a monitor program of the second scheduling platform to check the execution progress of the second scheduling task.
In this embodiment, if a flag file that the second scheduling task corresponding to the second scheduling task identifier runs successfully does not exist in the log table of the second scheduling platform, the first scheduling platform calls the monitor program of the second scheduling platform to check the execution progress of the second scheduling task. The task on the platform is to run, the mark of the dependent platform needs to be checked first, the mark is generated, the task is to run, and if the mark is not generated, the check needs to be continued for two minutes.
In one embodiment, the first and second scheduling platforms are any one of Oozie, linkdo, control-M, XXL-JOB or tivoliworkload scheduler.
The first scheduling platform and the second scheduling platform can be selected from any one of Oozie, linkdo, control-M, XXL-JOB and Tivoli Workload Scheduler as the second scheduling platform, and the second scheduling platform should not be identical to the first scheduling platform. The Oozie is a workflow engine server and is used for running the workflow of tasks such as hadoop map/reduce and hive, and meanwhile, the Oozie is also a java web program and is run in a java servlet container, for example, in a tomcat, the Oozie takes actions as a basic unit, and a plurality of actions can be formed into a mode operation of a DAG graph. Control-M adopts C/S mode, installs Enterprise Manager and server on server, installs agent on controlled host, agent can submit job flow defined by Control-M on host, and returns operation result. The JOB's operating conditions can be monitored throughout all batches, and JOB's operating conditions and process interventions can be controlled in a number of ways, as per Enterprise Manager. The distributed task scheduling platform XXL-JOB is an open source framework, is based on an open source Quartz scheduling kernel, is an integral scheduling task, cannot edit a large task flow chart, and tracks the execution state of each node. Dependency relationships cannot be configured among multiple tasks and the tasks run sequentially, for example, an upstream task signals a downstream task. Tivoli Workload Scheduler is a task scheduler software belonging to IBM that can correspond to cron under Unix, but it enhances many functions for enterprise scheduling, such as scheduling can be handled based on dependencies and event drivers, multi-time zone management, etc. cron can only schedule on a single server based on time, while TWS can replace cron to process job scheduling with its own daemon with richer functionality.
In one embodiment, the cross-platform scripting language is any one of Python, perl, ruby, PHP, lua or Java scripting language.
Scripting languages are executable files written in a format, also known as macros or batch files, using a particular descriptive language. Scripts may typically be invoked and executed temporarily by an application. Scripting languages have the advantages of rapid development, efficient execution, interpreted rather than compiled execution, and powerful communication functions between program components written in other languages. Wherein, any of the common scripting languages Python, perl, ruby, PHP, lua or Java may implement cross-scheduling platform communication.
As shown in fig. 2, in one embodiment, a dependency implementation apparatus across a scheduling platform is provided, the dependency implementation apparatus comprising:
the acquisition module is used for acquiring a first scheduling task identifier by the first scheduling platform;
the searching module is used for searching a second scheduling task identifier corresponding to the first scheduling task identifier in a first configuration table by the first scheduling platform, wherein the execution of the first scheduling task corresponding to the first scheduling task identifier depends on the execution of the second scheduling task corresponding to the second scheduling task identifier;
the polling module is used for polling whether a log table of the second scheduling platform has a mark file which is corresponding to the second scheduling task identifier and is successful in running or not by using a cross-platform scripting language by the first scheduling platform;
and the execution module is used for executing the first scheduling task corresponding to the first scheduling task identifier by the first scheduling platform if the mark file which is used for successfully running the second scheduling task corresponding to the second scheduling task identifier exists in the log table of the second scheduling platform.
In one embodiment, the cross-platform dependency implementation apparatus further includes a monitoring module, configured to, if a log table of a second scheduling platform does not have a flag file that the second scheduling task corresponding to the second scheduling task identifier runs successfully, call a monitor program of the second scheduling platform to check execution progress of the second scheduling task.
In one embodiment, the first scheduling platform and the second scheduling platform are any one of Oozie, linkdo, control-M, XXL-JOB or tivoliworkload scheduler platforms.
In one embodiment, the cross-platform scripting language is any one of Python, perl, ruby, PHP, lua or Java scripting language.
In one embodiment, as shown in FIG. 3, a computer device is presented that includes a processor, a non-volatile storage medium, a memory, and a network interface connected by a system bus. The nonvolatile storage medium of the computer device stores an operating system, a database and computer readable instructions, the database can store a control information sequence, and the computer readable instructions can enable the processor to inquire, download and verify the electronic invoice when the computer readable instructions are executed by the processor. The processor of the computer device is used to provide computing and control capabilities, supporting the operation of the entire computer device. The memory of the computer device may have stored therein computer readable instructions that, when executed by the processor, cause the processor to perform the steps of:
the first scheduling platform acquires a first scheduling task identifier;
the first scheduling platform searches a second scheduling task identifier corresponding to the first scheduling task identifier in a first configuration table, wherein the execution of the first scheduling task corresponding to the first scheduling task identifier depends on the execution of the second scheduling task corresponding to the second scheduling task identifier;
the first scheduling platform polls whether a log table of a second scheduling platform has a mark file which is corresponding to the second scheduling task identifier and is successful in running or not by using a cross-platform scripting language;
and if the log table of the second scheduling platform contains a mark file which is used for successfully operating the second scheduling task and corresponds to the second scheduling task identifier, the first scheduling platform executes the first scheduling task corresponding to the first scheduling task identifier.
In one embodiment, the method further comprises: and if the log table of the second scheduling platform does not contain a mark file which is corresponding to the second scheduling task identifier and is successful in running the second scheduling task, the first scheduling platform calls a monitoring program of the second scheduling platform to check the execution progress of the second scheduling task.
In one embodiment, the first scheduling task and the second scheduling task are controlled by detecting the state of a virtual task, wherein the virtual task is a running program which does not correspond to reality, only two states are prepared and completed, and the virtual task is uniformly reset to be in a ready state to wait for running.
In one embodiment, the first and second scheduling platforms are any one of Oozie, linkdo, control-M, XXL-JOB or tivoliworkload scheduler.
In one embodiment, the cross-platform scripting language is any one of Python, perl, ruby, PHP, lua or Java scripting language.
In one embodiment, a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
the first scheduling platform acquires a first scheduling task identifier;
the first scheduling platform searches a second scheduling task identifier corresponding to the first scheduling task identifier in a first configuration table, wherein the execution of the first scheduling task corresponding to the first scheduling task identifier depends on the execution of the second scheduling task corresponding to the second scheduling task identifier;
the first scheduling platform polls whether a log table of a second scheduling platform has a mark file which is corresponding to the second scheduling task identifier and is successful in running or not by using a cross-platform scripting language;
and if the log table of the second scheduling platform contains a mark file which is used for successfully operating the second scheduling task and corresponds to the second scheduling task identifier, the first scheduling platform executes the first scheduling task corresponding to the first scheduling task identifier.
In one embodiment, the method further comprises: and if the log table of the second scheduling platform does not contain a mark file which is corresponding to the second scheduling task identifier and is successful in running the second scheduling task, the first scheduling platform calls a monitoring program of the second scheduling platform to check the execution progress of the second scheduling task.
In one embodiment, the first and second scheduling platforms are any one of Oozie, linkdo, control-M, XXL-JOB or tivoliworkload scheduler.
In one embodiment, the cross-platform scripting language is any one of Python, perl, ruby, PHP, lua or Java scripting language.
Those of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be implemented by a program to instruct related hardware, the program may be stored in a computer readable storage medium, and the storage medium may include: read Only Memory (ROM), random access Memory (RAM, random Access Memory), magnetic or optical disk, and the like.
The technical features of the above-described embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above-described embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above-described embodiments represent only some exemplary embodiments of the invention, which are described in more detail and are not to be construed as limiting the scope of the invention. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the invention, which are all within the scope of the invention. Accordingly, the scope of protection of the present invention is to be determined by the appended claims.
Claims (10)
1. The method for realizing the dependence of the cross-dispatching platform is characterized by comprising the following steps:
the first scheduling platform acquires a first scheduling task identifier;
the first scheduling platform searches a second scheduling task identifier corresponding to the first scheduling task identifier in a first configuration table, wherein the execution of the first scheduling task corresponding to the first scheduling task identifier depends on the execution of the second scheduling task corresponding to the second scheduling task identifier;
the first scheduling platform polls whether a log table of a second scheduling platform has a mark file which is corresponding to the second scheduling task identifier and is successful in running or not by using a cross-platform scripting language;
and if the log table of the second scheduling platform contains a mark file which is used for successfully operating the second scheduling task and corresponds to the second scheduling task identifier, the first scheduling platform executes the first scheduling task corresponding to the first scheduling task identifier.
2. The cross-scheduling platform dependency implementation method of claim 1, further comprising:
and if the log table of the second scheduling platform does not contain a mark file which is corresponding to the second scheduling task identifier and is successful in running the second scheduling task, the first scheduling platform calls a monitoring program of the second scheduling platform to check the execution progress of the second scheduling task.
3. The method of claim 1, wherein the first scheduling platform and the second scheduling platform are any one of Oozie, linkdo, control-M, XXL-JOB or tivoliworkload scheduler.
4. The cross-platform dependency implementation method as claimed in claim 1, wherein the cross-platform scripting language is any one of Python, perl, ruby, PHP, lua or Java scripting language.
5. The dependency realization apparatus for cross-dispatching platform is characterized in that the dependency realization apparatus for cross-dispatching platform comprises:
the acquisition module is used for acquiring a first scheduling task identifier by the first scheduling platform;
the searching module is used for searching a second scheduling task identifier corresponding to the first scheduling task identifier in a first configuration table by the first scheduling platform, wherein the execution of the first scheduling task corresponding to the first scheduling task identifier depends on the execution of the second scheduling task corresponding to the second scheduling task identifier;
the polling module is used for polling whether a log table of the second scheduling platform has a mark file which is corresponding to the second scheduling task identifier and is successful in running or not by using a cross-platform scripting language by the first scheduling platform;
and the execution module is used for executing the first scheduling task corresponding to the first scheduling task identifier by the first scheduling platform if the mark file which is successful in running of the second scheduling task corresponding to the second scheduling task identifier exists in the log table of the second scheduling platform.
6. The cross-scheduling-platform dependency realization apparatus according to claim 5, further comprising a monitoring module, configured to, if a flag file for successful execution of a second scheduling task corresponding to the second scheduling task identifier does not exist in a log table of a second scheduling platform, call a monitor of the second scheduling platform to check execution progress of the second scheduling task by the first scheduling platform.
7. The cross-scheduling platform dependency realization apparatus of claim 5, wherein the first scheduling platform and the second scheduling platform are any one of Oozie, linkdo, control-M, XXL-JOB or tivoliworkload scheduler.
8. The cross-platform dependency realization apparatus of claim 5, wherein the cross-platform scripting language is any one of Python, perl, ruby, PHP, lua or Java scripting language.
9. A computer device comprising a memory and a processor, the memory having stored therein computer readable instructions which, when executed by the processor, cause the processor to perform the steps of the method of any of claims 1 to 4.
10. A storage medium storing computer readable instructions which, when executed by one or more processors, cause the one or more processors to perform the steps of the method of any of claims 1 to 4.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810486735.8A CN108710532B (en) | 2018-05-21 | 2018-05-21 | Dependency realization method, device, equipment and storage medium of cross-dispatching platform |
| PCT/CN2018/104708 WO2019223182A1 (en) | 2018-05-21 | 2018-09-08 | Cross-scheduling platform dependency implementation method and apparatus, device, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810486735.8A CN108710532B (en) | 2018-05-21 | 2018-05-21 | Dependency realization method, device, equipment and storage medium of cross-dispatching platform |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108710532A CN108710532A (en) | 2018-10-26 |
| CN108710532B true CN108710532B (en) | 2023-05-30 |
Family
ID=63868322
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810486735.8A Active CN108710532B (en) | 2018-05-21 | 2018-05-21 | Dependency realization method, device, equipment and storage medium of cross-dispatching platform |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN108710532B (en) |
| WO (1) | WO2019223182A1 (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110209893A (en) * | 2019-04-23 | 2019-09-06 | 北京奇艺世纪科技有限公司 | Task creating method, system and storage medium |
| CN110442392B (en) * | 2019-07-16 | 2022-08-09 | 新华三大数据技术有限公司 | Packet isolation method and device, electronic equipment and storage medium |
| CN110490451A (en) * | 2019-08-15 | 2019-11-22 | 中国平安财产保险股份有限公司 | Task data management-control method, device and computer equipment based on hadoop |
| CN110688101A (en) * | 2019-09-26 | 2020-01-14 | 山东浪潮通软信息科技有限公司 | Method and system for realizing distributed scheduling task based on XXL-JOB |
| CN111414264A (en) * | 2020-03-20 | 2020-07-14 | 北京奇艺世纪科技有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN111475272A (en) * | 2020-04-07 | 2020-07-31 | 四川虹美智能科技有限公司 | Method and device for controlling Java Web application timing task and task scheduling platform |
| CN111679899B (en) * | 2020-06-09 | 2023-05-23 | 微医云(杭州)控股有限公司 | Task scheduling method, device, platform equipment and storage medium |
| CN111984390A (en) * | 2020-08-31 | 2020-11-24 | 平安医疗健康管理股份有限公司 | Task scheduling method, device, equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107016479A (en) * | 2016-01-28 | 2017-08-04 | 五八同城信息技术有限公司 | Task scheduling and managing method, apparatus and system |
| CN107016480A (en) * | 2016-01-28 | 2017-08-04 | 五八同城信息技术有限公司 | Method for scheduling task, apparatus and system |
| CN107483649A (en) * | 2017-10-12 | 2017-12-15 | 福建富士通信息软件有限公司 | A kind of cross-platform log collection method based on Kaa services |
| CN107870807A (en) * | 2016-09-26 | 2018-04-03 | 平安科技（深圳）有限公司 | A kind of cross-platform method for scheduling task and device |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102096687B (en) * | 2009-12-14 | 2013-08-07 | 阿里巴巴集团控股有限公司 | Method and platform for scheduling tasks |
| US9652294B2 (en) * | 2013-11-25 | 2017-05-16 | International Business Machines Corporation | Cross-platform workload processing |
-
2018
- 2018-05-21 CN CN201810486735.8A patent/CN108710532B/en active Active
- 2018-09-08 WO PCT/CN2018/104708 patent/WO2019223182A1/en active Application Filing
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107016479A (en) * | 2016-01-28 | 2017-08-04 | 五八同城信息技术有限公司 | Task scheduling and managing method, apparatus and system |
| CN107016480A (en) * | 2016-01-28 | 2017-08-04 | 五八同城信息技术有限公司 | Method for scheduling task, apparatus and system |
| CN107870807A (en) * | 2016-09-26 | 2018-04-03 | 平安科技（深圳）有限公司 | A kind of cross-platform method for scheduling task and device |
| CN107483649A (en) * | 2017-10-12 | 2017-12-15 | 福建富士通信息软件有限公司 | A kind of cross-platform log collection method based on Kaa services |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108710532A (en) | 2018-10-26 |
| WO2019223182A1 (en) | 2019-11-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108710532B (en) | Dependency realization method, device, equipment and storage medium of cross-dispatching platform | |
| CN108287694B (en) | Application program construction method, system, computer device and storage medium | |
| US8412984B2 (en) | Debugging in a cluster processing network | |
| CN108399132B (en) | Scheduling test method, device and storage medium | |
| US8146060B2 (en) | Data processing system and method for execution of a test routine in connection with an operating system | |
| CN107016480B (en) | Task scheduling method, device and system | |
| US20170132026A1 (en) | Apparatus and method for optimizing startup of embedded system | |
| US7802302B1 (en) | Single scan for a base machine and all associated virtual machines | |
| CN111324423B (en) | Method and device for monitoring processes in container, storage medium and computer equipment | |
| US11861505B2 (en) | Method and apparatus of executing dynamic graph for neural network computation | |
| CN110162344B (en) | Isolation current limiting method and device, computer equipment and readable storage medium | |
| CN113220431B (en) | Cross-cloud distributed data task scheduling method, device and storage medium | |
| US11868829B2 (en) | System and method for the remote execution of one or more arbitrarily defined workflows | |
| US20220326927A1 (en) | Abort installation of firmware bundles | |
| CN113256296A (en) | Intelligent contract execution method, system, device and storage medium | |
| CN114546588A (en) | Task deployment method and device, storage medium and electronic device | |
| CN112328602B (en) | Method, device and equipment for writing data into Kafka | |
| CN116339908A (en) | Virtual machine starting method, device, computer equipment and storage medium | |
| CN110674024A (en) | Electronic equipment integration test system and method thereof | |
| CN113792026B (en) | Method and device for deploying database script and computer-readable storage medium | |
| CN110109747B (en) | Apache Spark-based data exchange method, system and server | |
| CN112416394B (en) | Service upgrading method and device, storage medium and electronic equipment | |
| CN112130900B (en) | User information management method, system, equipment and medium for BMC | |
| CN113553098A (en) | Method and device for submitting Flink SQL (structured query language) operation and computer equipment | |
| Vani | Building, deploying and validating a home location register (HLR) using Jenkins under the docker and container environment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |