WO2019223182A1 - Cross-scheduling platform dependency implementation method and apparatus, device, and storage medium - Google Patents

Cross-scheduling platform dependency implementation method and apparatus, device, and storage medium Download PDF

Info

Publication number
WO2019223182A1
WO2019223182A1 PCT/CN2018/104708 CN2018104708W WO2019223182A1 WO 2019223182 A1 WO2019223182 A1 WO 2019223182A1 CN 2018104708 W CN2018104708 W CN 2018104708W WO 2019223182 A1 WO2019223182 A1 WO 2019223182A1
Authority
WO
WIPO (PCT)
Prior art keywords
scheduling
platform
scheduling task
task
task identifier
Prior art date
Application number
PCT/CN2018/104708
Other languages
French (fr)
Chinese (zh)
Inventor
刘斌
袁贺强
Original Assignee
平安科技(深圳)有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 平安科技(深圳)有限公司 filed Critical 平安科技(深圳)有限公司
Publication of WO2019223182A1 publication Critical patent/WO2019223182A1/en

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/48Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806Task transfer initiation or dispatching
    • G06F9/4843Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00Energy efficient computing, e.g. low power processors, power management or thermal management

Definitions

  • the present application relates to the field of computer technology, and in particular, to a method, an apparatus, a device, and a storage medium for implementing a dependency across a scheduling platform.
  • OOZIE is a server based on a workflow engine
  • Data exists in hadoop HIVE (Hadoop, Hadoop Distributed File System, HDFS for short), which is a distributed system infrastructure.
  • Hive is a A data warehouse architecture built on the Hadoop file system, and analyzes and manages the data stored in HDFS) table;
  • linkdo is a distributed task scheduling platform
  • the data of the hadoop task and the project B task depends on the generation of the data in the HIVE table in the project A. If there is no dependency, it can only be achieved by staggering the execution time. For example, the task of project A runs at 5 o'clock and is expected to be completed in 1-3 hours, then the task of project B can only be set to be executed after 8 o'clock.
  • project B Because the running time of the task fluctuates daily, if the task of project A is completed in 1 hour, then project B will be set to execute at 8 o'clock, and it will wait for two hours; or when the task of project B is executed, A The project's mission data has not yet been fully generated.
  • a cross-platform platform dependency implementation method includes:
  • the first scheduling platform acquires a first scheduling task identifier
  • the first scheduling platform searches a first configuration table for a second scheduling task identifier corresponding to the first scheduling task identifier, wherein execution of the first scheduling task corresponding to the first scheduling task identifier depends on the Execution of the second scheduled task corresponding to the second scheduled task identifier;
  • the first scheduling platform uses a cross-platform scripting language to poll a log table of the second scheduling platform whether a flag file corresponding to the second scheduling task identifier runs successfully;
  • the first scheduling platform executes the first scheduling task corresponding to the first scheduling task identifier Scheduling tasks.
  • a dependency implementation device across a scheduling platform includes:
  • An obtaining module configured to obtain a first scheduling task identifier by the first scheduling platform
  • the searching module is configured to search the first scheduling platform for a second scheduling task identifier corresponding to the first scheduling task identifier in a first configuration table, where the first scheduling task corresponding to the first scheduling task identifier is The execution depends on the execution of the second scheduling task corresponding to the second scheduling task identifier;
  • a polling module configured to use a cross-platform scripting language for the first scheduling platform to poll a log table of the second scheduling platform whether a flag file corresponding to the second scheduling task identifier runs successfully;
  • the execution module is configured to: if a log file of the second scheduling task corresponding to the second scheduling task identifier exists in the log table of the second scheduling platform, the first scheduling platform executes the first scheduling task. Identifies the corresponding first scheduled task.
  • a computer device includes a memory and a processor.
  • the memory stores computer-readable instructions.
  • the processor causes the processor to perform the following steps:
  • the first scheduling platform acquires a first scheduling task identifier
  • the first scheduling platform searches a first configuration table for a second scheduling task identifier corresponding to the first scheduling task identifier, wherein execution of the first scheduling task corresponding to the first scheduling task identifier depends on the Execution of the second scheduled task corresponding to the second scheduled task identifier;
  • the first scheduling platform uses a cross-platform scripting language to poll a log table of the second scheduling platform whether a flag file corresponding to the second scheduling task identifier runs successfully;
  • the first scheduling platform executes the first scheduling task corresponding to the first scheduling task identifier Scheduling tasks.
  • a storage medium storing computer-readable instructions.
  • the one or more processors execute the following steps:
  • the first scheduling platform acquires a first scheduling task identifier
  • the first scheduling platform searches a first configuration table for a second scheduling task identifier corresponding to the first scheduling task identifier, wherein execution of the first scheduling task corresponding to the first scheduling task identifier depends on the Execution of the second scheduled task corresponding to the second scheduled task identifier;
  • the first scheduling platform uses a cross-platform scripting language to poll a log table of the second scheduling platform whether a flag file corresponding to the second scheduling task identifier runs successfully;
  • the first scheduling platform executes the first scheduling task corresponding to the first scheduling task identifier Scheduling tasks.
  • the above-mentioned dependent implementation method, device, device and storage medium of the scheduling platform obtains the first scheduling task identifier through the first scheduling platform, and the first scheduling platform looks up the first configuration table corresponding to the first scheduling task identifier The second scheduling task identifier, wherein the execution of the first scheduling task corresponding to the first scheduling task identifier depends on the execution of the second scheduling task corresponding to the second scheduling task identifier, and the first scheduling platform uses Platform scripting language, polling the log table of the second scheduling platform for a flag file corresponding to the second scheduling task identifier for the successful running of the second scheduling task, if the log table of the second scheduling platform exists with the second scheduling platform A flag file indicating that the second scheduling task corresponding to the scheduling task identifier runs successfully, the first scheduling platform executes the first scheduling task corresponding to the first scheduling task identifier, and if the log table of the second scheduling platform does not exist with the A flag file corresponding to the second scheduling task identifier that the second scheduling task runs successfully, the first calling platform calls
  • FIG. 1 is a flowchart of a method for implementing a dependency across a scheduling platform in an embodiment
  • FIG. 2 is a structural block diagram of a device for implementing a dependency across a scheduling platform in an embodiment
  • FIG. 3 is a block diagram of the internal structure of a computer device in one embodiment.
  • a dependency implementation method across a scheduling platform includes:
  • Step S101 The first scheduling platform obtains a first scheduling task identifier; the first scheduling platform is a distributed scheduling platform, and may be any one of Oozie, Linkdo, Control-M, XXL-JOB, and TivoliWorkloadScheduler; among them,
  • the distributed task scheduling platform XXL-JOB is an open source framework based on the open source Quartz scheduling kernel. Scheduling tasks are a whole. You cannot edit the flowchart of a large task and track the execution status of each node. You cannot configure dependencies between multiple tasks and run them sequentially, such as upstream tasks sending signals to downstream tasks.
  • IBM's Tivoli Workload Scheduler can correspond to cron under Unix, but it enhances a lot of functions for enterprise scheduling, such as can handle scheduling based on dependencies and event-driven, can manage multiple time zones, and so on. Cron can only schedule based on time on a single server, while TWS can replace cron and use its own background program to handle job scheduling with richer functions.
  • Oozie is a workflow engine server that runs task workflows such as hadoop map / reduce and hive.
  • Control-M using C / S mode, installs the Enterprise Manager and server on the server, and installs the agent on the controlled host.
  • the agent can submit the job stream defined by Control-M on the host and return the running result.
  • Control-M On the Enterprise Manager, you can comprehensively monitor the operation of all batch jobs. There are multiple ways to control the operating conditions and process intervention of JOB.
  • the first scheduling task identifier is set to uniquely distinguish a scheduling task to be executed on the first scheduling platform, and may be a combination of letters, numbers, and underscores of a preset rule.
  • the specific implementation steps for the first scheduling platform to obtain the first scheduling task may be: the first scheduling platform sequentially scans the current status of each task in a preset scheduling task table, and if the current status is ready or ready Status, obtain the ready status or the scheduled task ID corresponding to the ready status.
  • the scheduling task table sets scheduling task information such as the scheduling task name, the scheduling task identifier, and the scheduling task status.
  • the scheduling task status is set to represent the current status of the scheduling task, including the ready status / ready status and completion status. It can be understood that, in this embodiment, the scheduling task information is stored separately.
  • the scheduling task information may be stored in the same task table as the non-scheduled task information of the first scheduling platform (that is, the task information executed by the first scheduling platform itself without the need to implement execution by calling other platforms).
  • the specific implementation steps for the first scheduling platform to obtain the first scheduling task may be: the first scheduling platform reads the task type of each task in turn in a preset task table, and if the task type is a scheduling task, obtaining the scheduling task If the current state is the ready state or the ready state, obtain the scheduling task identifier corresponding to the ready state or the ready state.
  • Step S102 the first scheduling platform searches a first configuration table for a second scheduling task identifier corresponding to the first scheduling task identifier, wherein the execution dependency of the first scheduling task corresponding to the first scheduling task identifier is Execution of a second scheduling task corresponding to the second scheduling task identifier;
  • the second calling platform is a called platform that corresponds to the first scheduling platform and uses a different framework from the first scheduling platform.
  • the first scheduling platform uses a platform of the XXL-JOB framework, so the second calling platform may use Oozie. , Linkdo, Control-M, or TivoliWorkloadScheduler framework platform;
  • the second scheduling platform identifier is set to uniquely distinguish the scheduling task to be executed on the second scheduling platform, can also be preset rules of letters, numbers and underlined A combination string;
  • the first configuration table is set on the first scheduling platform and stores a mapping relationship between the first scheduling task task identifier and the second platform task identifier, wherein the first scheduling task corresponding to the first scheduling task identifier The execution depends on the execution of the second scheduling task corresponding to the second scheduling task identifier, that is, after the second scheduling task corresponding to the second scheduling platform identifier is successfully executed, the first scheduling task corresponding to the first scheduling platform identifier is executed.
  • the following table is an example of setting the fields of the first configuration table in an embodiment.
  • the first scheduling platform uses the platform of the XXL-JOB framework
  • the second calling platform uses the platform of the Oozie framework.
  • oozie_task_name is the first scheduled task identifier
  • the field task_name is the second scheduled task identifier:
  • Step S103 The first scheduling platform uses a cross-platform scripting language to poll a log table of the second scheduling platform whether a flag file corresponding to the second scheduling task identifier runs successfully;
  • the log table is set in advance on the second scheduling platform, and is set to receive a flag file that the second scheduling task runs successfully. After the second scheduling platform executes the second scheduling task, it generates a flag file according to the running status of the scheduling task, and stores the flag file in the created log table.
  • the following table is an example of a log table in an embodiment, where RUN-DATA is the running data, OOZIE-TASK-NAME is the OOZIE task name, TASK-NAME is the task name, and CHECK-TIME is the check time.
  • a cross-platform scripting language is invoked through the crontab of LINUX.
  • One of Python, Perl, Ruby, PHP, Lua, and Java can be selected as a cross-platform scripting language.
  • the selected cross-platform scripting language is triggered once, and the cross-platform scripting language utilizes its detection function to use the cross-platform scripting language on the first scheduling platform every one minute to poll the log table of the second scheduling platform for existence with the second scheduling
  • step S104 if a log file of the second scheduling task corresponding to the second scheduling task identifier exists in the log table of the second scheduling platform, the first scheduling platform executes corresponding to the first scheduling task identifier. The first scheduled task.
  • the first scheduling platform executes the first scheduling corresponding to the first scheduling task identifier task.
  • the first scheduling platform is the active side
  • the second scheduling platform is the passive side.
  • the first scheduling platform initiates a query to query the flag in the second scheduling platform. If the flag is completed, the first scheduling platform will trigger his own task to start running to achieve the purpose of dependence.
  • the mark can be agreed, it can be a file with time and task tags, or it can be a field in the database.
  • the first calling platform invokes the monitoring of the second platform.
  • the first calling platform invokes a monitoring program of the second platform to view the Implementation progress of the second scheduling task. If the task on the platform wants to run, you need to first check the flag of the dependent platform. If the flag is generated, then run it. If the flag is not generated, you need to continue the inspection after two minutes.
  • the first scheduling platform and the second scheduling platform are any one of Oozie, Linkdo, Control-M, XXL-JOB, or TivoliWorkloadScheduler.
  • the first scheduling platform and the second scheduling platform can be selected from Oozie, Linkdo, Control-M, XXL-JOB, and Tivoli Workload Scheduler as the second scheduling platform.
  • the second scheduling platform should be the same as the first scheduling platform. Can't be the same.
  • Oozie is a workflow engine server for running tasks such as hadoop map / reduce and hive.
  • Oozie is also a java web program running in a java servlet container, such as in tomcat.
  • Oozie uses action as the basic unit. Multiple actions form a DAG diagram. Control-M, using C / S mode, installs the Enterprise Manager and server on the server, and installs the agent on the controlled host.
  • the agent can submit the job stream defined by Control-M on the host and return the running result.
  • Control-M On the Enterprise Manager, you can comprehensively monitor the operation of all batch jobs.
  • the distributed task scheduling platform XXL-JOB is an open source framework based on the open source Quartz scheduling kernel. Scheduling tasks are a whole. You cannot edit the flowchart of a large task and track the execution status of each node. You cannot configure dependencies between multiple tasks and run them sequentially, such as upstream tasks sending signals to downstream tasks.
  • Tivoli Workload Scheduler is a task scheduling software belonging to IBM, which can correspond to cron under Unix, but it has enhanced a lot of functions for enterprise scheduling, such as can handle scheduling based on dependencies and event-driven, can manage multiple time zones, and so on. Cron can only schedule based on time on a single server, while TWS can replace cron and use its own background program to handle job scheduling with richer functions.
  • the cross-platform scripting language is any one of Python, Perl, Ruby, PHP, Lua, or Java.
  • a scripting language is an executable file written in a certain format using a specific descriptive language, also known as a macro or batch file. Scripts can usually be called and executed temporarily by an application. Scripting languages have rapid development, efficient execution, interpretation rather than compilation and execution, and powerful communication capabilities with program components written in other languages. Among them, any of the common scripting languages Python, Perl, Ruby, PHP, Lua, or Java can implement communication across scheduling platforms.
  • a dependency implementation device across a scheduling platform includes:
  • An obtaining module configured to obtain a first scheduling task identifier by the first scheduling platform
  • the searching module is configured to search the first scheduling platform for a second scheduling task identifier corresponding to the first scheduling task identifier in a first configuration table, where the first scheduling task corresponding to the first scheduling task identifier is The execution depends on the execution of the second scheduling task corresponding to the second scheduling task identifier;
  • a polling module configured to use a cross-platform scripting language for the first scheduling platform to poll a log table of the second scheduling platform whether a flag file corresponding to the second scheduling task identifier runs successfully;
  • the execution module is configured to: if a log file of the second scheduling task corresponding to the second scheduling task identifier exists in the log table of the second scheduling platform, the first scheduling platform executes the first scheduling task. Identifies the corresponding first scheduled task.
  • the cross-platform dependency implementation device further includes a monitoring module configured to set a flag indicating that the second scheduling task corresponding to the second scheduling task identifier runs successfully if no log table of the second scheduling platform exists.
  • File the first calling platform calls a monitoring program of the second platform to view the execution progress of the second scheduling task.
  • the first scheduling platform and the second scheduling platform are any one of Oozie, Linkdo, Control-M, XXL-JOB, or TivoliWorkloadScheduler.
  • the cross-platform scripting language is any one of Python, Perl, Ruby, PHP, Lua, or Java.
  • the acquisition module includes:
  • a scanning unit configured to scan the current status of each task in turn by the first scheduling platform in a preset scheduling task table
  • the determining unit is configured to obtain a scheduling task identifier corresponding to the ready state or the ready state if the current state is a ready state or a ready state, and the scheduling task identifier is set to the first scheduling task identifier.
  • a computer device in one embodiment, as shown in FIG. 3, a computer device is proposed.
  • the computer device includes a processor, a nonvolatile storage medium, a memory, and a network interface connected through a system bus.
  • the non-volatile storage medium of the computer device stores an operating system, a database, and computer-readable instructions.
  • the database may store control information sequences.
  • the processor may enable the processor to query. , Download and verify electronic invoices.
  • the processor of the computer device is configured to provide computing and control capabilities to support the operation of the entire computer device.
  • the memory of the computer device may store computer-readable instructions.
  • the processor may cause the processor to execute the steps in the method for implementing a dependency across a scheduling platform in the foregoing embodiments.
  • a storage medium storing computer-readable instructions.
  • the computer-readable instructions are executed by one or more processors, the one or more processors are caused to perform cross-scheduling in the foregoing embodiments. Platform-dependent implementation steps.
  • the storage medium may be a non-volatile storage medium.
  • the program may be stored in a computer-readable storage medium.
  • the storage medium may include: Read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disks or optical disks, etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Debugging And Monitoring (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Stored Programmes (AREA)

Abstract

A cross-scheduling platform dependency implementation method and apparatus, a device, and a storage medium. The method comprises: a first scheduling platform acquiring a first scheduling task identifier (S101); the first scheduling platform searching in a first configuration table for a second scheduling task identifier corresponding to the first scheduling task identifier, the execution of a first scheduling task corresponding to the first scheduling task identifier depending on the execution of a second scheduling task corresponding to the second scheduling task identifier (S102); the first scheduling platform using a cross-platform script language to poll regarding whether a log table of a second scheduling platform has a flag file that shows the second scheduling task corresponding to the second scheduling task identifier is successfully operated (S103); if the log table of the second scheduling platform has a flag file that shows the second scheduling task corresponding to the second scheduling task identifier is successfully operated, the first scheduling platform executing the first scheduling task corresponding to the first scheduling task identifier (S104). Said method facilitates searching for and monitoring the operation result of a scheduling task, implementing the dependency of tasks between scheduling platforms.

Description

跨调度平台的依赖实现方法、装置、设备和存储介质Method, device, equipment and storage medium for implementing dependency across scheduling platforms
本申请要求于2018年05月21日提交中国专利局、申请号为201810486735.8、发明名称为“跨调度平台的依赖实现方法、装置、设备和存储介质”的中国专利申请的优先权,其全部内容通过引用结合在本申请中。This application claims the priority of a Chinese patent application filed on May 21, 2018 with the Chinese Patent Office, application number 201810486735.8, and invention name "Cross-platform platform dependent implementation method, device, device, and storage medium", its entire content Incorporated by reference in this application.
技术领域Technical field
本申请涉及计算机技术领域,尤其涉及一种跨调度平台的依赖实现方法、装置、设备和存储介质。The present application relates to the field of computer technology, and in particular, to a method, an apparatus, a device, and a storage medium for implementing a dependency across a scheduling platform.
背景技术Background technique
目前大数据任务运行种类繁多,且在不同调度平台上运行,由于不同调度平台任务运行的方式和规则不一样,无法直接进行数据的依赖,从而无法形成有效严谨的数据流,开发人员只能通过定时去实现依赖,容易出现数据不完整和等待时间过长等问题。At present, there are many types of big data tasks running on different scheduling platforms. Due to the different methods and rules of tasks running on different scheduling platforms, it is not possible to directly rely on data, which can not form an effective and rigorous data flow. Timing to implement dependencies is prone to problems such as incomplete data and long waiting times.
例如,有一个项目A,使用OOZIE(OOZIE是一个基于工作流引擎的服务器)来调度任务,数据存在hadoop HIVE(Hadoop,Hadoop Distributed File System,简称HDFS,是一个分布式系统基础架构,Hive是一种建立在Hadoop文件系统上的数据仓库架构,并对存储在HDFS中的数据进行分析与管理)表中;另有一个项目B,采用的是linkdo(linkdo是一种分布式任务调度平台)调度hadoop任务,项目B任务的数据要依赖项目A中HIVE表中的数据的生成。如果不形成依赖,只能通过错开执行时间的方式去实现,譬如,A项目的任务5点跑,预计1-3个小时执行完,那么B项目的任务就只能设置在8点后执行。For example, there is a project A that uses OOZIE (OOZIE is a server based on a workflow engine) to schedule tasks. Data exists in hadoop HIVE (Hadoop, Hadoop Distributed File System, HDFS for short), which is a distributed system infrastructure. Hive is a A data warehouse architecture built on the Hadoop file system, and analyzes and manages the data stored in HDFS) table; another project B uses linkdo (linkdo is a distributed task scheduling platform) scheduling The data of the hadoop task and the project B task depends on the generation of the data in the HIVE table in the project A. If there is no dependency, it can only be achieved by staggering the execution time. For example, the task of project A runs at 5 o'clock and is expected to be completed in 1-3 hours, then the task of project B can only be set to be executed after 8 o'clock.
因为任务的运行时间每日有波动,如果A项目任务1个小时就完成了,那么B项目因为设置了在8点执行,就会多等待两个小时;或者B项目的任务执行的时候,A项目的任务的数据还未完全产生。Because the running time of the task fluctuates daily, if the task of project A is completed in 1 hour, then project B will be set to execute at 8 o'clock, and it will wait for two hours; or when the task of project B is executed, A The project's mission data has not yet been fully generated.
发明内容Summary of the Invention
基于此,有必要针对现有技术容易造成任务运行时间冗长,数据出错的问题,提供一种跨调度平台的依赖实现方法、装置、设备和存储介质。Based on this, it is necessary to provide a method, an apparatus, a device, and a storage medium for implementing a dependency across a scheduling platform in response to the problems that the existing technology is apt to cause long task running time and data error.
一种跨调度平台的依赖实现方法,包括:A cross-platform platform dependency implementation method includes:
第一调度平台获取第一调度任务标识;The first scheduling platform acquires a first scheduling task identifier;
所述第一调度平台在第一配置表中查找与所述第一调度任务标识对应的第二调度任务标识,其中,所述第一调度任务标识对应的第一调度任务的执行依赖于所述第二调度任务标识对应的第二调度任务的执行;The first scheduling platform searches a first configuration table for a second scheduling task identifier corresponding to the first scheduling task identifier, wherein execution of the first scheduling task corresponding to the first scheduling task identifier depends on the Execution of the second scheduled task corresponding to the second scheduled task identifier;
所述第一调度平台利用跨平台脚本语言,轮询第二调度平台的日志表是否存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件;The first scheduling platform uses a cross-platform scripting language to poll a log table of the second scheduling platform whether a flag file corresponding to the second scheduling task identifier runs successfully;
若第二调度平台的日志表中存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件,则所述第一调度平台执行与所述第一调度任务标识对应的第一调度任务。If a log file of the second scheduling task corresponding to the second scheduling task identifier exists in the log table of the second scheduling platform, the first scheduling platform executes the first scheduling task corresponding to the first scheduling task identifier Scheduling tasks.
一种跨调度平台的依赖实现装置,包括:A dependency implementation device across a scheduling platform includes:
获取模块,设置为第一调度平台获取第一调度任务标识;An obtaining module, configured to obtain a first scheduling task identifier by the first scheduling platform;
查找模块,设置为所述第一调度平台在第一配置表中查找与所述第一调度任务标识对应的第二调度任务标识,其中,所述第一调度任务标识对应的第一调度任务的执行依赖于所述第二调度任务标识对应的第二调度任务的执行;The searching module is configured to search the first scheduling platform for a second scheduling task identifier corresponding to the first scheduling task identifier in a first configuration table, where the first scheduling task corresponding to the first scheduling task identifier is The execution depends on the execution of the second scheduling task corresponding to the second scheduling task identifier;
轮询模块,设置为所述第一调度平台利用跨平台脚本语言,轮询第二调度平台的日志表是否存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件;A polling module, configured to use a cross-platform scripting language for the first scheduling platform to poll a log table of the second scheduling platform whether a flag file corresponding to the second scheduling task identifier runs successfully;
执行模块,设置为若第二调度平台的日志表中存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件,则所述第一调度平台执行与所述第一调度任务标识对应的第一调度任务。The execution module is configured to: if a log file of the second scheduling task corresponding to the second scheduling task identifier exists in the log table of the second scheduling platform, the first scheduling platform executes the first scheduling task. Identifies the corresponding first scheduled task.
一种计算机设备,包括存储器和处理器,所述存储器中存储有计算机可读 指令,所述计算机可读指令被所述处理器执行时,使得所述处理器执行以下步骤:A computer device includes a memory and a processor. The memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor causes the processor to perform the following steps:
第一调度平台获取第一调度任务标识;The first scheduling platform acquires a first scheduling task identifier;
所述第一调度平台在第一配置表中查找与所述第一调度任务标识对应的第二调度任务标识,其中,所述第一调度任务标识对应的第一调度任务的执行依赖于所述第二调度任务标识对应的第二调度任务的执行;The first scheduling platform searches a first configuration table for a second scheduling task identifier corresponding to the first scheduling task identifier, wherein execution of the first scheduling task corresponding to the first scheduling task identifier depends on the Execution of the second scheduled task corresponding to the second scheduled task identifier;
所述第一调度平台利用跨平台脚本语言,轮询第二调度平台的日志表是否存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件;The first scheduling platform uses a cross-platform scripting language to poll a log table of the second scheduling platform whether a flag file corresponding to the second scheduling task identifier runs successfully;
若第二调度平台的日志表中存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件,则所述第一调度平台执行与所述第一调度任务标识对应的第一调度任务。If a log file of the second scheduling task corresponding to the second scheduling task identifier exists in the log table of the second scheduling platform, the first scheduling platform executes the first scheduling task corresponding to the first scheduling task identifier Scheduling tasks.
一种存储有计算机可读指令的存储介质,所述计算机可读指令被一个或多个处理器执行时,使得一个或多个处理器执行以下步骤:A storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the following steps:
第一调度平台获取第一调度任务标识;The first scheduling platform acquires a first scheduling task identifier;
所述第一调度平台在第一配置表中查找与所述第一调度任务标识对应的第二调度任务标识,其中,所述第一调度任务标识对应的第一调度任务的执行依赖于所述第二调度任务标识对应的第二调度任务的执行;The first scheduling platform searches a first configuration table for a second scheduling task identifier corresponding to the first scheduling task identifier, wherein execution of the first scheduling task corresponding to the first scheduling task identifier depends on the Execution of the second scheduled task corresponding to the second scheduled task identifier;
所述第一调度平台利用跨平台脚本语言,轮询第二调度平台的日志表是否存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件;The first scheduling platform uses a cross-platform scripting language to poll a log table of the second scheduling platform whether a flag file corresponding to the second scheduling task identifier runs successfully;
若第二调度平台的日志表中存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件,则所述第一调度平台执行与所述第一调度任务标识对应的第一调度任务。If a log file of the second scheduling task corresponding to the second scheduling task identifier exists in the log table of the second scheduling platform, the first scheduling platform executes the first scheduling task corresponding to the first scheduling task identifier Scheduling tasks.
上述跨调度平台的依赖实现方法、装置、设备和存储介质,通过第一调度平台获取第一调度任务标识,所述第一调度平台在第一配置表中查找与所述第一调度任务标识对应的第二调度任务标识,其中,所述第一调度任务标识对应的第一调度任务的执行依赖于所述第二调度任务标识对应的第二调度任务的执 行,所述第一调度平台利用跨平台脚本语言,轮询第二调度平台的日志表是否存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件,若第二调度平台的日志表中存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件,则所述第一调度平台执行与所述第一调度任务标识对应的第一调度任务,若第二调度平台的日志表中不存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件,则所述第一调用平台调用第二平台的监控程序以查看所述第二调度任务的执行进展,实现了调度平台之间的依赖,避免了定时依赖出现的数据不完整,实现了调度平台之间的融合贯通。The above-mentioned dependent implementation method, device, device and storage medium of the scheduling platform obtains the first scheduling task identifier through the first scheduling platform, and the first scheduling platform looks up the first configuration table corresponding to the first scheduling task identifier The second scheduling task identifier, wherein the execution of the first scheduling task corresponding to the first scheduling task identifier depends on the execution of the second scheduling task corresponding to the second scheduling task identifier, and the first scheduling platform uses Platform scripting language, polling the log table of the second scheduling platform for a flag file corresponding to the second scheduling task identifier for the successful running of the second scheduling task, if the log table of the second scheduling platform exists with the second scheduling platform A flag file indicating that the second scheduling task corresponding to the scheduling task identifier runs successfully, the first scheduling platform executes the first scheduling task corresponding to the first scheduling task identifier, and if the log table of the second scheduling platform does not exist with the A flag file corresponding to the second scheduling task identifier that the second scheduling task runs successfully, the first calling platform calls The monitoring program of the two platforms checks the execution progress of the second scheduling task, realizes the dependency between the scheduling platforms, avoids the incomplete data appearing in the timing dependency, and realizes the integration and integration between the scheduling platforms.
附图说明BRIEF DESCRIPTION OF THE DRAWINGS
通过阅读下文优选实施方式的详细描述,各种其他的优点和益处对于本领域普通技术人员将变得清楚明了。附图仅用于示出优选实施方式的目的,而并不认为是对本申请的限制。Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the detailed description of the preferred embodiments below. The drawings are only for the purpose of illustrating preferred embodiments and are not to be considered as limiting the present application.
图1为一个实施例中跨调度平台的依赖实现方法的流程图;FIG. 1 is a flowchart of a method for implementing a dependency across a scheduling platform in an embodiment; FIG.
图2为一个实施例中跨调度平台的依赖实现装置的结构框图;2 is a structural block diagram of a device for implementing a dependency across a scheduling platform in an embodiment;
图3为一个实施例中计算机设备的内部结构框图。FIG. 3 is a block diagram of the internal structure of a computer device in one embodiment.
具体实施方式Detailed ways
为了使本申请的目的、技术方案及优点更加清楚明白,以下结合附图及实施例,对本申请进行进一步详细说明。应当理解,此处所描述的具体实施例仅仅用以解释本申请,并不用于限定本申请。In order to make the purpose, technical solution, and advantages of the present application clearer, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application, and are not used to limit the application.
本技术领域技术人员可以理解,除非特意声明,这里使用的单数形式“一”、“一个”、“所述”和“该”也可包括复数形式。应该进一步理解的是,本申请的说明书中使用的措辞“包括”是指存在所述特征、整数、步骤、操作、元件和/或组件,但是并不排除存在或添加一个或多个其他特征、整数、步骤、操作、 元件、组件和/或它们的组。Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms "a", "an", "the" and "the" may include plural forms. It should be further understood that the word "comprising" used in the specification of the present application refers to the presence of the described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, Integers, steps, operations, elements, components, and / or groups thereof.
作为一个较好的实施例,如图1所示,一种跨调度平台的依赖实现方法,该依赖实现方法包括:As a better embodiment, as shown in FIG. 1, a dependency implementation method across a scheduling platform includes:
步骤S101,第一调度平台获取第一调度任务标识;第一调度平台为分布式调度平台,可以是Oozie、Linkdo、Control-M、XXL-JOB、TivoliWorkloadScheduler中的任一种平台;其中,Step S101: The first scheduling platform obtains a first scheduling task identifier; the first scheduling platform is a distributed scheduling platform, and may be any one of Oozie, Linkdo, Control-M, XXL-JOB, and TivoliWorkloadScheduler; among them,
分布式任务调度平台XXL-JOB是一个开源框架,基于开源Quartz调度内核,调度任务是一个整体,不能编辑一个大任务的流程图,并跟踪每个节点的执行状态。多个任务之间,无法配置依赖关系、依次运行,比如上游任务给下游任务发信号。IBM的Tivoli Workload Scheduler,可以对应Unix下的cron,但它增强了很多对于企业调度的功能,如可以基于依赖关系和事件驱动来处理调度,可以对多时区管理等等。cron仅仅能够在单台服务器上基于时间来调度,而TWS可以取代cron,用自己的后台程序以更加丰富的功能来处理作业调度安排。Oozie是一个工作流引擎服务器,用于运行hadoop map/reduce和hive等任务工作流。Control-M,采用C/S模式,在服务器上安装Enterprise Manager和服务器,在被控主机上安装agent,agent可以在主机上提交由Control-M定义好的作业流,并返回运行结果。在Enterprise Manager上面则可以全面监视所有批量作业的运行情况,可以有多种方式控制JOB的运行条件和过程干预。The distributed task scheduling platform XXL-JOB is an open source framework based on the open source Quartz scheduling kernel. Scheduling tasks are a whole. You cannot edit the flowchart of a large task and track the execution status of each node. You cannot configure dependencies between multiple tasks and run them sequentially, such as upstream tasks sending signals to downstream tasks. IBM's Tivoli Workload Scheduler can correspond to cron under Unix, but it enhances a lot of functions for enterprise scheduling, such as can handle scheduling based on dependencies and event-driven, can manage multiple time zones, and so on. Cron can only schedule based on time on a single server, while TWS can replace cron and use its own background program to handle job scheduling with richer functions. Oozie is a workflow engine server that runs task workflows such as hadoop map / reduce and hive. Control-M, using C / S mode, installs the Enterprise Manager and server on the server, and installs the agent on the controlled host. The agent can submit the job stream defined by Control-M on the host and return the running result. On the Enterprise Manager, you can comprehensively monitor the operation of all batch jobs. There are multiple ways to control the operating conditions and process intervention of JOB.
第一调度任务标识设置为唯一区分第一调度平台上待执行的调度任务,可以是预设规则的字母、数字及下划线的组合字符串。本实施例中,第一调度平台获取第一调度任务的具体实施步骤可以是:第一调度平台在预先设置的调度任务表中依次扫描每一个任务的当前状态,若当前状态为准备状态或就绪状态,则获取准备状态或就绪状态对应的调度任务标识。其中,调度任务表中设置有调度任务名称,调度任务标识,调度任务状态等调度任务信息;调度任务状态设置为表征调度任务的当前状态,包括准备状态/就绪状态和完成状态。可以理解的是,本实施例中,调度任务信息是单独存储的。The first scheduling task identifier is set to uniquely distinguish a scheduling task to be executed on the first scheduling platform, and may be a combination of letters, numbers, and underscores of a preset rule. In this embodiment, the specific implementation steps for the first scheduling platform to obtain the first scheduling task may be: the first scheduling platform sequentially scans the current status of each task in a preset scheduling task table, and if the current status is ready or ready Status, obtain the ready status or the scheduled task ID corresponding to the ready status. The scheduling task table sets scheduling task information such as the scheduling task name, the scheduling task identifier, and the scheduling task status. The scheduling task status is set to represent the current status of the scheduling task, including the ready status / ready status and completion status. It can be understood that, in this embodiment, the scheduling task information is stored separately.
在其他实施例中,调度任务信息可以和第一调度平台的非调度任务信息(即第一调度平台自己执行而不需要通过调用其他平台实现执行的任务信息)存储在同一张任务表中,那么,第一调度平台获取第一调度任务的具体实施步骤可以是:第一调度平台在预先设置的任务表中依次读取每一个任务的任务类型,若任务类型是调度任务,则获取该调度任务的当前状态,若当前状态为准备状态或就绪状态,则获取准备状态或就绪状态对应的调度任务标识。In other embodiments, the scheduling task information may be stored in the same task table as the non-scheduled task information of the first scheduling platform (that is, the task information executed by the first scheduling platform itself without the need to implement execution by calling other platforms). The specific implementation steps for the first scheduling platform to obtain the first scheduling task may be: the first scheduling platform reads the task type of each task in turn in a preset task table, and if the task type is a scheduling task, obtaining the scheduling task If the current state is the ready state or the ready state, obtain the scheduling task identifier corresponding to the ready state or the ready state.
步骤S102,所述第一调度平台在第一配置表中查找与所述第一调度任务标识对应的第二调度任务标识,其中,所述第一调度任务标识对应的第一调度任务的执行依赖于所述第二调度任务标识对应的第二调度任务的执行;Step S102, the first scheduling platform searches a first configuration table for a second scheduling task identifier corresponding to the first scheduling task identifier, wherein the execution dependency of the first scheduling task corresponding to the first scheduling task identifier is Execution of a second scheduling task corresponding to the second scheduling task identifier;
第二调用平台是对应于第一调度平台,且与第一调度平台采用不同框架的被调用平台,例如第一调度平台采用的是XXL-JOB框架的平台,那么第二调用平台可以是采用Oozie、Linkdo、Control-M或TivoliWorkloadScheduler中的任一种框架的平台;第二调度平台标识设置为唯一区分第二调度平台上待执行的调度任务,同样可以是预设规则的字母、数字及下划线的组合字符串;第一配置表设置在第一调度平台,且存储有第一调度平台任务标识与第二平台任务标识之间的映射关系,其中,第一调度任务标识对应的第一调度任务的执行依赖于第二调度任务标识对应的第二调度任务的执行,即第二调度平台标识对应的第二调度任务成功执行后,第一调度平台标识对应的第一调度任务才被执行。The second calling platform is a called platform that corresponds to the first scheduling platform and uses a different framework from the first scheduling platform. For example, the first scheduling platform uses a platform of the XXL-JOB framework, so the second calling platform may use Oozie. , Linkdo, Control-M, or TivoliWorkloadScheduler framework platform; the second scheduling platform identifier is set to uniquely distinguish the scheduling task to be executed on the second scheduling platform, can also be preset rules of letters, numbers and underlined A combination string; the first configuration table is set on the first scheduling platform and stores a mapping relationship between the first scheduling task task identifier and the second platform task identifier, wherein the first scheduling task corresponding to the first scheduling task identifier The execution depends on the execution of the second scheduling task corresponding to the second scheduling task identifier, that is, after the second scheduling task corresponding to the second scheduling platform identifier is successfully executed, the first scheduling task corresponding to the first scheduling platform identifier is executed.
下表为一个实施例中第一配置表的字段设置示例,本实施例中,第一调度平台采用的是XXL-JOB框架的平台,第二调用平台采用的是Oozie框架的平台,其中,字段oozie_task_name是第一调度任务标识,字段task_name是第二调度任务标识:The following table is an example of setting the fields of the first configuration table in an embodiment. In this embodiment, the first scheduling platform uses the platform of the XXL-JOB framework, and the second calling platform uses the platform of the Oozie framework. oozie_task_name is the first scheduled task identifier, and the field task_name is the second scheduled task identifier:
表1Table 1
Figure PCTCN2018104708-appb-000001
Figure PCTCN2018104708-appb-000001
Figure PCTCN2018104708-appb-000002
Figure PCTCN2018104708-appb-000002
步骤S103,所述第一调度平台利用跨平台脚本语言,轮询第二调度平台的日志表是否存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件;Step S103: The first scheduling platform uses a cross-platform scripting language to poll a log table of the second scheduling platform whether a flag file corresponding to the second scheduling task identifier runs successfully;
日志表预先设置在第二调度平台上,设置为接收第二调度任务运行成功的标志文件。第二调度平台执行第二调度任务后,会根据调度任务的运行情况产生一个标志文件,并将该标志文件存放于所创建的日志表中。The log table is set in advance on the second scheduling platform, and is set to receive a flag file that the second scheduling task runs successfully. After the second scheduling platform executes the second scheduling task, it generates a flag file according to the running status of the scheduling task, and stores the flag file in the created log table.
下表为一个实施例中日志表的示例,其中,RUN-DATA是运行数据,OOZIE-TASK-NAME是OOZIE任务名称,TASK-NAME是任务名称,CHECK-TIME是检查时间。The following table is an example of a log table in an embodiment, where RUN-DATA is the running data, OOZIE-TASK-NAME is the OOZIE task name, TASK-NAME is the task name, and CHECK-TIME is the check time.
表2Table 2
Figure PCTCN2018104708-appb-000003
Figure PCTCN2018104708-appb-000003
本实施例中,通过LINUX的crontab调用一跨平台脚本语言,可以从Python,Perl,Ruby,PHP,Lua,Java中任选一种作为跨平台脚本语言,通过LINUX的crontab功能,每隔一分钟触发一次所选跨平台脚本语言,该跨平台脚本语言利用其检测功能每隔一分钟在第一调度平台利用跨平台脚本语言,轮询第二调度平台的日志表是否存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件,若存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件则将调度平台中的虚拟任务置为完成状态。In this embodiment, a cross-platform scripting language is invoked through the crontab of LINUX. One of Python, Perl, Ruby, PHP, Lua, and Java can be selected as a cross-platform scripting language. Through the crontab function of Linux, every one minute The selected cross-platform scripting language is triggered once, and the cross-platform scripting language utilizes its detection function to use the cross-platform scripting language on the first scheduling platform every one minute to poll the log table of the second scheduling platform for existence with the second scheduling The flag file corresponding to the successful execution of the second scheduling task corresponding to the task identifier, and if a flag file corresponding to the second scheduling task identifier to successfully run exists, the virtual task in the scheduling platform is set to a completed state.
步骤S104,若第二调度平台的日志表中存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件,则所述第一调度平台执行与所述第一调度任务标识对应的第一调度任务。In step S104, if a log file of the second scheduling task corresponding to the second scheduling task identifier exists in the log table of the second scheduling platform, the first scheduling platform executes corresponding to the first scheduling task identifier. The first scheduled task.
本实施例中,若第二调度平台的日志表中存在与第二调度任务标识对应的第二调度任务运行成功的标志文件,则第一调度平台执行与第一调度任务标识对应的第一调度任务。第一调度平台是主动方,第二调度平台是被动方。第一调度平台发起查询,查询第二调度平台中的标志,如果标志完成了,第一调度平台就会触发他自己的任务开始运行,达到依赖的目的。所述标志可以约定,可以是某个带有时间和任务标签的文件也可以是数据库中的某个字段。In this embodiment, if a log file of the second scheduling task corresponding to the second scheduling task identifier exists in the log table of the second scheduling platform, the first scheduling platform executes the first scheduling corresponding to the first scheduling task identifier task. The first scheduling platform is the active side, and the second scheduling platform is the passive side. The first scheduling platform initiates a query to query the flag in the second scheduling platform. If the flag is completed, the first scheduling platform will trigger his own task to start running to achieve the purpose of dependence. The mark can be agreed, it can be a file with time and task tags, or it can be a field in the database.
在一个实施例中,若第二调度平台的日志表中不存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件,则所述第一调用平台调用第二平台的监控程序以查看所述第二调度任务的执行进展。In one embodiment, if there is no flag file corresponding to the second scheduling task identifier in the log table of the second scheduling platform, the first calling platform invokes the monitoring of the second platform. A program to view the execution progress of the second scheduled task.
本实施例中,若第二调度平台的日志表中不存在与第二调度任务标识对应的第二调度任务运行成功的标志文件,则第一调用平台调用第二平台的监控程序以查看所述第二调度任务的执行进展。平台上的任务想要运行起来,需要先去检查依赖的这个平台的标志,标志产生了,那就运行,如果标志没有产生,就需要过两分钟继续检查。In this embodiment, if a log file of the second scheduling task corresponding to the second scheduling task identifier does not exist in the log table of the second scheduling platform, the first calling platform invokes a monitoring program of the second platform to view the Implementation progress of the second scheduling task. If the task on the platform wants to run, you need to first check the flag of the dependent platform. If the flag is generated, then run it. If the flag is not generated, you need to continue the inspection after two minutes.
在一个实施例中,所述第一调度平台和第二调度平台为Oozie、Linkdo、Control-M、XXL-JOB或TivoliWorkloadScheduler中的任一种平台。In one embodiment, the first scheduling platform and the second scheduling platform are any one of Oozie, Linkdo, Control-M, XXL-JOB, or TivoliWorkloadScheduler.
第一个调度平台和第二调度平台,可从Oozie、Linkdo、Control-M、XXL-JOB、Tivoli Workload Scheduler中选任一种平台作为第二调度平台,该第二调度平台应当与第一调度平台不能相同。Oozie是一个工作流引擎服务器,用于运行hadoop map/reduce和hive等任务工作流,同时Oozie还是一个java web程序,运行在java servlet容器中,如tomcat中,Oozie以action为基本单位,可以将多个action构成一个DAG图的模式运行。Control-M,采用C/S模式,在服务器上安装Enterprise Manager和服务器,在被控主机上安装agent,agent可以在主机上提交由Control-M定义好的作业流,并返回运行结果。在Enterprise Manager上面则可以全面监视所有批量作业的运行情况,可以有多种方式控制JOB的运行条件和过程干预。分布式任务调度平台XXL-JOB是一个开源框架,基于开源Quartz调度内核,调度任务是一个整体,不能编辑一个大任务的流程图,并跟踪每个节点的执行状态。多个任务之间,无法配置依赖关系、依次运行,比如上游任务给下游任务发信号。Tivoli Workload Scheduler是属于IBM的一个任务调度软件,可以对应Unix下的cron,但它增强了很多对于企业调度的功能,如可以基于依赖关系和事件驱动来处理调度,可以对多时区管理等等。cron仅仅能够在单台服务器上基于时间来调度,而TWS可以取代cron,用自己的后台程序以更加丰富的功能来处理作业调度安排。The first scheduling platform and the second scheduling platform can be selected from Oozie, Linkdo, Control-M, XXL-JOB, and Tivoli Workload Scheduler as the second scheduling platform. The second scheduling platform should be the same as the first scheduling platform. Can't be the same. Oozie is a workflow engine server for running tasks such as hadoop map / reduce and hive. At the same time, Oozie is also a java web program running in a java servlet container, such as in tomcat. Oozie uses action as the basic unit. Multiple actions form a DAG diagram. Control-M, using C / S mode, installs the Enterprise Manager and server on the server, and installs the agent on the controlled host. The agent can submit the job stream defined by Control-M on the host and return the running result. On the Enterprise Manager, you can comprehensively monitor the operation of all batch jobs. There are multiple ways to control the operating conditions and process intervention of JOB. The distributed task scheduling platform XXL-JOB is an open source framework based on the open source Quartz scheduling kernel. Scheduling tasks are a whole. You cannot edit the flowchart of a large task and track the execution status of each node. You cannot configure dependencies between multiple tasks and run them sequentially, such as upstream tasks sending signals to downstream tasks. Tivoli Workload Scheduler is a task scheduling software belonging to IBM, which can correspond to cron under Unix, but it has enhanced a lot of functions for enterprise scheduling, such as can handle scheduling based on dependencies and event-driven, can manage multiple time zones, and so on. Cron can only schedule based on time on a single server, while TWS can replace cron and use its own background program to handle job scheduling with richer functions.
在一个实施例中,跨平台脚本语言为Python、Perl、Ruby、PHP、Lua或Java中任一种脚本语言。In one embodiment, the cross-platform scripting language is any one of Python, Perl, Ruby, PHP, Lua, or Java.
脚本语言是使用一种特定的描述性语言,依据一定的格式编写的可执行文件,又称作宏或批处理文件。脚本通常可以由应用程序临时调用并执行。脚本语言具有快速开发,高效率的执行,解释而非编译执行,和其它语言编写的程序组件之间通信功能很强大。其中,常见的脚本语言Python、Perl、Ruby、PHP、Lua或Java中任一种脚本语言都可以实现跨调度平台的通信。A scripting language is an executable file written in a certain format using a specific descriptive language, also known as a macro or batch file. Scripts can usually be called and executed temporarily by an application. Scripting languages have rapid development, efficient execution, interpretation rather than compilation and execution, and powerful communication capabilities with program components written in other languages. Among them, any of the common scripting languages Python, Perl, Ruby, PHP, Lua, or Java can implement communication across scheduling platforms.
如图2所示,在一个实施例中,提供一种跨调度平台的依赖实现装置,该依赖实现装置包括:As shown in FIG. 2, in one embodiment, a dependency implementation device across a scheduling platform is provided. The dependency implementation device includes:
获取模块,设置为第一调度平台获取第一调度任务标识;An obtaining module, configured to obtain a first scheduling task identifier by the first scheduling platform;
查找模块,设置为所述第一调度平台在第一配置表中查找与所述第一调度任务标识对应的第二调度任务标识,其中,所述第一调度任务标识对应的第一调度任务的执行依赖于所述第二调度任务标识对应的第二调度任务的执行;The searching module is configured to search the first scheduling platform for a second scheduling task identifier corresponding to the first scheduling task identifier in a first configuration table, where the first scheduling task corresponding to the first scheduling task identifier is The execution depends on the execution of the second scheduling task corresponding to the second scheduling task identifier;
轮询模块,设置为所述第一调度平台利用跨平台脚本语言,轮询第二调度平台的日志表是否存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件;A polling module, configured to use a cross-platform scripting language for the first scheduling platform to poll a log table of the second scheduling platform whether a flag file corresponding to the second scheduling task identifier runs successfully;
执行模块,设置为若第二调度平台的日志表中存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件,则所述第一调度平台执行与所述第一调度任务标识对应的第一调度任务。The execution module is configured to: if a log file of the second scheduling task corresponding to the second scheduling task identifier exists in the log table of the second scheduling platform, the first scheduling platform executes the first scheduling task. Identifies the corresponding first scheduled task.
在一个实施例中,所述跨平台的依赖实现装置还包括监控模块,设置为若第二调度平台的日志表中不存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件,则所述第一调用平台调用第二平台的监控程序以查看所述第二调度任务的执行进展。In an embodiment, the cross-platform dependency implementation device further includes a monitoring module configured to set a flag indicating that the second scheduling task corresponding to the second scheduling task identifier runs successfully if no log table of the second scheduling platform exists. File, the first calling platform calls a monitoring program of the second platform to view the execution progress of the second scheduling task.
在一个实施例中,述第一调度平台和所述第二调度平台为Oozie、Linkdo、Control-M、XXL-JOB或TivoliWorkloadScheduler中的任一种平台。In one embodiment, the first scheduling platform and the second scheduling platform are any one of Oozie, Linkdo, Control-M, XXL-JOB, or TivoliWorkloadScheduler.
在一个实施例中,所述跨平台脚本语言为Python、Perl、Ruby、PHP、Lua或Java中任一种脚本语言。In one embodiment, the cross-platform scripting language is any one of Python, Perl, Ruby, PHP, Lua, or Java.
在一个实施例中,所述获取模块包括:In one embodiment, the acquisition module includes:
扫描单元,设置为所述第一调度平台在预先设置的调度任务表中依次扫描每一个任务的当前状态;A scanning unit, configured to scan the current status of each task in turn by the first scheduling platform in a preset scheduling task table;
判断单元,设置为若所述当前状态为准备状态或就绪状态,则获取所述准备状态或所述就绪状态对应的调度任务标识,所述调度任务标识设置为所述第一调度任务标识。The determining unit is configured to obtain a scheduling task identifier corresponding to the ready state or the ready state if the current state is a ready state or a ready state, and the scheduling task identifier is set to the first scheduling task identifier.
在一个实施例中,如图3所示,提出了一种计算机设备,该计算机设备包括通过系统总线连接的处理器、非易失性存储介质、存储器和网络接口。其中,该计算机设备的非易失性存储介质存储有操作系统、数据库和计算机可读指令,数据库中可存储有控件信息序列,该计算机可读指令被处理器执行时,可使得处理器实现查询、下载和验证电子发票。该计算机设备的处理器设置为提供计算和控制能力,支撑整个计算机设备的运行。该计算机设备的存储器中可存储有计算机可读指令,该计算机可读指令被处理器执行时,可使得处理器执行上述各实施例里跨调度平台的依赖实现方法中的步骤。In one embodiment, as shown in FIG. 3, a computer device is proposed. The computer device includes a processor, a nonvolatile storage medium, a memory, and a network interface connected through a system bus. The non-volatile storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store control information sequences. When the computer-readable instructions are executed by the processor, the processor may enable the processor to query. , Download and verify electronic invoices. The processor of the computer device is configured to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor may cause the processor to execute the steps in the method for implementing a dependency across a scheduling platform in the foregoing embodiments.
在一个实施例中,提出了一种存储有计算机可读指令的存储介质,该计算机可读指令被一个或多个处理器执行时,使得一个或多个处理器执行上述各实施例里跨调度平台的依赖实现方法中的步骤。其中,存储介质可以为非易失性存储介质。In one embodiment, a storage medium storing computer-readable instructions is provided. When the computer-readable instructions are executed by one or more processors, the one or more processors are caused to perform cross-scheduling in the foregoing embodiments. Platform-dependent implementation steps. The storage medium may be a non-volatile storage medium.
本领域普通技术人员可以理解上述实施例的各种方法中的全部或部分步骤是可以通过程序来指令相关的硬件来完成,该程序可以存储于一计算机可读存储介质中,存储介质可以包括:只读存储器(ROM,Read Only Memory)、随机存取存储器(RAM,Random Access Memory)、磁盘或光盘等。A person of ordinary skill in the art may understand that all or part of the steps in the various methods of the foregoing embodiments may be implemented by a program instructing related hardware. The program may be stored in a computer-readable storage medium. The storage medium may include: Read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disks or optical disks, etc.
以上所述实施例的各技术特征可以进行任意的组合,为使描述简洁,未对上述实施例中的各个技术特征所有可能的组合都进行描述,然而,只要这些技术特征的组合不存在矛盾,都应当认为是本说明书记载的范围。The technical features of the embodiments described above can be arbitrarily combined. In order to simplify the description, all possible combinations of the technical features in the above embodiments have not been described. However, as long as there is no contradiction in the combination of these technical features, It should be considered as the scope described in this specification.
以上所述实施例仅表达了本申请一些示例性实施例,其描述较为具体和详细,但并不能因此而理解为对本申请专利范围的限制。应当指出的是,对于本领域的普通技术人员来说,在不脱离本申请构思的前提下,还可以做出若干变形和改进,这些都属于本申请的保护范围。因此,本申请专利的保护范围应以所附权利要求为准。The above-mentioned embodiments only express some exemplary embodiments of the present application, and their descriptions are more specific and detailed, but cannot be understood as a limitation on the scope of the patent of the present application. It should be noted that, for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of this application patent shall be subject to the appended claims.

Claims (20)

  1. 一种跨调度平台的依赖实现方法,包括:A cross-platform platform dependency implementation method includes:
    第一调度平台获取第一调度任务标识;The first scheduling platform acquires a first scheduling task identifier;
    所述第一调度平台在第一配置表中查找与所述第一调度任务标识对应的第二调度任务标识,其中,所述第一调度任务标识对应的第一调度任务的执行依赖于所述第二调度任务标识对应的第二调度任务的执行;The first scheduling platform searches a first configuration table for a second scheduling task identifier corresponding to the first scheduling task identifier, wherein execution of the first scheduling task corresponding to the first scheduling task identifier depends on the Execution of the second scheduled task corresponding to the second scheduled task identifier;
    所述第一调度平台利用跨平台脚本语言,轮询第二调度平台的日志表是否存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件;The first scheduling platform uses a cross-platform scripting language to poll a log table of the second scheduling platform whether a flag file corresponding to the second scheduling task identifier runs successfully;
    若第二调度平台的日志表中存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件,则所述第一调度平台执行与所述第一调度任务标识对应的第一调度任务。If a log file of the second scheduling task corresponding to the second scheduling task identifier exists in the log table of the second scheduling platform, the first scheduling platform executes the first scheduling task corresponding to the first scheduling task identifier Scheduling tasks.
  2. 根据权利要求1所述的跨调度平台的依赖实现方法,其中,所述方法还包括:The method for implementing a cross-platform platform dependency according to claim 1, wherein the method further comprises:
    若第二调度平台的日志表中不存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件,则所述第一调用平台调用第二平台的监控程序以查看所述第二调度任务的执行进展。If a log file of the second scheduling task corresponding to the second scheduling task identifier does not exist in the log table of the second scheduling platform, the first calling platform invokes a monitoring program of the second platform to view the first scheduling task. The progress of the implementation of the second scheduling task.
  3. 根据权利要求1所述的跨调度平台的依赖实现方法,其中,所述第一调度平台和所述第二调度平台为Oozie、Linkdo、Control-M、XXL-JOB或TivoliWorkloadScheduler中的任一种平台。The cross-platform platform dependent implementation method according to claim 1, wherein the first scheduling platform and the second scheduling platform are any one of Oozie, Linkdo, Control-M, XXL-JOB, or TivoliWorkloadScheduler .
  4. 根据权利要求1所述的跨调度平台的依赖实现方法,其中,所述跨平台脚本语言为Python、Perl、Ruby、PHP、Lua或Java中任一种脚本语言。The cross-platform platform dependent implementation method according to claim 1, wherein the cross-platform scripting language is any one of Python, Perl, Ruby, PHP, Lua, or Java.
  5. 根据权利要求1所述的跨调度平台的依赖实现方法,其中,所述第一调度平台获取第一调度任务标识包括:所述第一调度平台在预先设置的调度任务表中依次扫描每一个任务的当前状态,若所述当前状态为准备状态或就绪状态,则获取所述准备状态或所述就绪状态对应的调度任务标识,所述调度任务标识设置为所述第一调度任务标识。The method for implementing a dependency across a scheduling platform according to claim 1, wherein the first scheduling platform obtaining the first scheduling task identifier comprises: the first scheduling platform sequentially scanning each task in a preset scheduling task table If the current state is a ready state or a ready state, obtaining a scheduling task identifier corresponding to the ready state or the ready state, and the scheduling task identifier is set to the first scheduling task identifier.
  6. 一种跨调度平台的依赖实现装置,包括:A dependency implementation device across a scheduling platform includes:
    获取模块,设置为第一调度平台获取第一调度任务标识;An obtaining module, configured to obtain a first scheduling task identifier by the first scheduling platform;
    查找模块,设置为所述第一调度平台在第一配置表中查找与所述第一调度 任务标识对应的第二调度任务标识,其中,所述第一调度任务标识对应的第一调度任务的执行依赖于所述第二调度任务标识对应的第二调度任务的执行;The searching module is configured to search the first scheduling platform for a second scheduling task identifier corresponding to the first scheduling task identifier in a first configuration table, where the first scheduling task corresponding to the first scheduling task identifier is The execution depends on the execution of the second scheduling task corresponding to the second scheduling task identifier;
    轮询模块,设置为所述第一调度平台利用跨平台脚本语言,轮询第二调度平台的日志表是否存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件;A polling module, configured to use a cross-platform scripting language for the first scheduling platform to poll a log table of the second scheduling platform whether a flag file corresponding to the second scheduling task identifier runs successfully;
    执行模块,设置为若第二调度平台的日志表中存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件,则所述第一调度平台执行与所述第一调度任务标识对应的第一调度任务。The execution module is configured to: if a log file of the second scheduling task corresponding to the second scheduling task identifier exists in the log table of the second scheduling platform, the first scheduling platform executes the first scheduling task. Identifies the corresponding first scheduled task.
  7. 根据权利要求6所述的跨调度平台的依赖实现装置,其中,还包括监控模块,设置为若第二调度平台的日志表中不存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件,则所述第一调用平台调用第二平台的监控程序以查看所述第二调度任务的执行进展。The device according to claim 6, further comprising a monitoring module configured to run if a second scheduling task corresponding to the second scheduling task identifier does not exist in a log table of the second scheduling platform If the flag file is successful, the first calling platform invokes a monitoring program of the second platform to view the execution progress of the second scheduling task.
  8. 根据权利要求6所述的跨调度平台的依赖实现装置,其中,所述第一调度平台和所述第二调度平台为Oozie、Linkdo、Control-M、XXL-JOB或TivoliWorkloadScheduler中的任一种平台。The dependent implementation device for a cross-scheduling platform according to claim 6, wherein the first scheduling platform and the second scheduling platform are any one of Oozie, Linkdo, Control-M, XXL-JOB, or TivoliWorkloadScheduler .
  9. 根据权利要求6所述的跨调度平台的依赖实现装置,其中,所述跨平台脚本语言为Python、Perl、Ruby、PHP、Lua或Java中任一种脚本语言。The dependent implementation device for a cross-scheduling platform according to claim 6, wherein the cross-platform scripting language is any one of Python, Perl, Ruby, PHP, Lua, or Java.
  10. 根据权利要求6所述的跨调度平台的依赖实现装置,其中,所述获取模块包括:The device for implementing a dependency of a cross-scheduling platform according to claim 6, wherein the acquisition module comprises:
    扫描单元,设置为所述第一调度平台在预先设置的调度任务表中依次扫描每一个任务的当前状态;A scanning unit, configured to scan the current status of each task in turn by the first scheduling platform in a preset scheduling task table;
    判断单元,设置为若所述当前状态为准备状态或就绪状态,则获取所述准备状态或所述就绪状态对应的调度任务标识,所述调度任务标识设置为所述第一调度任务标识。The determining unit is configured to obtain a scheduling task identifier corresponding to the ready state or the ready state if the current state is a ready state or a ready state, and the scheduling task identifier is set to the first scheduling task identifier.
  11. 一种计算机设备,其中,包括存储器和处理器,所述存储器中存储有计算机可读指令,所述计算机可读指令被所述处理器执行时,使得所述处理器执行以下步骤:A computer device includes a memory and a processor. The memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor causes the processor to perform the following steps:
    第一调度平台获取第一调度任务标识;The first scheduling platform acquires a first scheduling task identifier;
    所述第一调度平台在第一配置表中查找与所述第一调度任务标识对应的第二调度任务标识,其中,所述第一调度任务标识对应的第一调度任务的执行依赖于所述第二调度任务标识对应的第二调度任务的执行;The first scheduling platform searches a first configuration table for a second scheduling task identifier corresponding to the first scheduling task identifier, wherein execution of the first scheduling task corresponding to the first scheduling task identifier depends on the Execution of the second scheduled task corresponding to the second scheduled task identifier;
    所述第一调度平台利用跨平台脚本语言,轮询第二调度平台的日志表是否存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件;The first scheduling platform uses a cross-platform scripting language to poll a log table of the second scheduling platform whether a flag file corresponding to the second scheduling task identifier runs successfully;
    若第二调度平台的日志表中存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件,则所述第一调度平台执行与所述第一调度任务标识对应的第一调度任务。If a log file of the second scheduling task corresponding to the second scheduling task identifier exists in the log table of the second scheduling platform, the first scheduling platform executes the first scheduling task corresponding to the first scheduling task identifier Scheduling tasks.
  12. 根据权利要求11所述的计算机设备,其中,所述计算机可读指令被所述处理器执行时,还使得所述处理器执行以下步骤:The computer device according to claim 11, wherein the computer-readable instructions, when executed by the processor, further cause the processor to perform the following steps:
    若第二调度平台的日志表中不存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件,则所述第一调用平台调用第二平台的监控程序以查看所述第二调度任务的执行进展。If a log file of the second scheduling task corresponding to the second scheduling task identifier does not exist in the log table of the second scheduling platform, the first calling platform invokes a monitoring program of the second platform to view the first scheduling task. The progress of the implementation of the second scheduling task.
  13. 根据权利要求11所述的计算机设备,其中,所述第一调度平台和所述第二调度平台为Oozie、Linkdo、Control-M、XXL-JOB或TivoliWorkloadScheduler中的任一种平台。The computer device according to claim 11, wherein the first scheduling platform and the second scheduling platform are any one of Oozie, Linkdo, Control-M, XXL-JOB, or TivoliWorkloadScheduler.
  14. 根据权利要求11所述的计算机设备,其中,所述跨平台脚本语言为Python、Perl、Ruby、PHP、Lua或Java中任一种脚本语言。The computer device according to claim 11, wherein the cross-platform scripting language is any one of Python, Perl, Ruby, PHP, Lua, or Java.
  15. 根据权利要求11所述的计算机设备,其中,所述第一调度平台获取第一调度任务标识,使得所述处理器执行以下步骤:The computer device according to claim 11, wherein the first scheduling platform obtains a first scheduling task identifier, so that the processor performs the following steps:
    所述第一调度平台在预先设置的调度任务表中依次扫描每一个任务的当前状态,若所述当前状态为准备状态或就绪状态,则获取所述准备状态或所述就绪状态对应的调度任务标识,所述调度任务标识设置为所述第一调度任务标识。The first scheduling platform sequentially scans the current status of each task in a preset scheduling task table, and if the current status is a ready state or a ready state, obtaining the scheduled task corresponding to the ready state or the ready state An identifier, and the scheduling task identifier is set to the first scheduling task identifier.
  16. 一种存储有计算机可读指令的存储介质,其中,所述计算机可读指令被一个或多个处理器执行时,使得一个或多个处理器执行以下步骤:A storage medium storing computer-readable instructions, wherein when the computer-readable instructions are executed by one or more processors, the one or more processors execute the following steps:
    第一调度平台获取第一调度任务标识;The first scheduling platform acquires a first scheduling task identifier;
    所述第一调度平台在第一配置表中查找与所述第一调度任务标识对应的第二调度任务标识,其中,所述第一调度任务标识对应的第一调度任务的执行依 赖于所述第二调度任务标识对应的第二调度任务的执行;The first scheduling platform searches a first configuration table for a second scheduling task identifier corresponding to the first scheduling task identifier, wherein execution of the first scheduling task corresponding to the first scheduling task identifier depends on the Execution of the second scheduled task corresponding to the second scheduled task identifier;
    所述第一调度平台利用跨平台脚本语言,轮询第二调度平台的日志表是否存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件;The first scheduling platform uses a cross-platform scripting language to poll a log table of the second scheduling platform whether a flag file corresponding to the second scheduling task identifier runs successfully;
    若第二调度平台的日志表中存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件,则所述第一调度平台执行与所述第一调度任务标识对应的第一调度任务。If a log file of the second scheduling task corresponding to the second scheduling task identifier exists in the log table of the second scheduling platform, the first scheduling platform executes the first scheduling task corresponding to the first scheduling task identifier Scheduling tasks.
  17. 根据权利要求16所述的存储介质,其中,所述计算机可读指令被一个或多个处理器执行时,还使得一个或多个处理器执行以下步骤:The storage medium of claim 16, wherein the computer-readable instructions, when executed by one or more processors, further cause the one or more processors to perform the following steps:
    若第二调度平台的日志表中不存在与所述第二调度任务标识对应的第二调度任务运行成功的标志文件,则所述第一调用平台调用第二平台的监控程序以查看所述第二调度任务的执行进展。If a log file of the second scheduling task corresponding to the second scheduling task identifier does not exist in the log table of the second scheduling platform, the first calling platform invokes a monitoring program of the second platform to view the first scheduling task. The progress of the implementation of the second scheduling task.
  18. 根据权利要求16所述的存储介质,其中,所述第一调度平台和所述第二调度平台为Oozie、Linkdo、Control-M、XXL-JOB或TivoliWorkloadScheduler中的任一种平台。The storage medium according to claim 16, wherein the first scheduling platform and the second scheduling platform are any one of Oozie, Linkdo, Control-M, XXL-JOB, or TivoliWorkloadScheduler.
  19. 根据权利要求16所述的存储介质,其中,所述跨平台脚本语言为Python、Perl、Ruby、PHP、Lua或Java中任一种脚本语言。The storage medium according to claim 16, wherein the cross-platform scripting language is any one of Python, Perl, Ruby, PHP, Lua, or Java.
  20. 根据权利要求16所述的存储介质,其中,所述第一调度平台获取第一调度任务标识,使得一个或多个处理器执行以下步骤:The storage medium according to claim 16, wherein the first scheduling platform obtains a first scheduling task identifier, so that one or more processors perform the following steps:
    所述第一调度平台在预先设置的调度任务表中依次扫描每一个任务的当前状态,若所述当前状态为准备状态或就绪状态,则获取所述准备状态或所述就绪状态对应的调度任务标识,所述调度任务标识设置为所述第一调度任务标识。The first scheduling platform sequentially scans the current status of each task in a preset scheduling task table, and if the current status is a ready state or a ready state, obtaining the scheduled task corresponding to the ready state or the ready state An identifier, and the scheduling task identifier is set to the first scheduling task identifier.
PCT/CN2018/104708 2018-05-21 2018-09-08 Cross-scheduling platform dependency implementation method and apparatus, device, and storage medium WO2019223182A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810486735.8A CN108710532B (en) 2018-05-21 2018-05-21 Dependency realization method, device, equipment and storage medium of cross-dispatching platform
CN201810486735.8 2018-05-21

Publications (1)

Publication Number Publication Date
WO2019223182A1 true WO2019223182A1 (en) 2019-11-28

Family

ID=63868322

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/104708 WO2019223182A1 (en) 2018-05-21 2018-09-08 Cross-scheduling platform dependency implementation method and apparatus, device, and storage medium

Country Status (2)

Country Link
CN (1) CN108710532B (en)
WO (1) WO2019223182A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111984390A (en) * 2020-08-31 2020-11-24 平安医疗健康管理股份有限公司 Task scheduling method, device, equipment and storage medium

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109814983A (en) * 2019-01-02 2019-05-28 深圳壹账通智能科技有限公司 Task batch processing method, device, computer equipment and storage medium
CN110209893A (en) * 2019-04-23 2019-09-06 北京奇艺世纪科技有限公司 Task creating method, system and storage medium
CN110442392B (en) * 2019-07-16 2022-08-09 新华三大数据技术有限公司 Packet isolation method and device, electronic equipment and storage medium
CN110490451A (en) * 2019-08-15 2019-11-22 中国平安财产保险股份有限公司 Task data management-control method, device and computer equipment based on hadoop
CN110688101A (en) * 2019-09-26 2020-01-14 山东浪潮通软信息科技有限公司 Method and system for realizing distributed scheduling task based on XXL-JOB
CN111414264A (en) * 2020-03-20 2020-07-14 北京奇艺世纪科技有限公司 Data processing method and device, electronic equipment and storage medium
CN111475272A (en) * 2020-04-07 2020-07-31 四川虹美智能科技有限公司 Method and device for controlling Java Web application timing task and task scheduling platform
CN111679899B (en) * 2020-06-09 2023-05-23 微医云(杭州)控股有限公司 Task scheduling method, device, platform equipment and storage medium
CN113835848A (en) * 2020-06-24 2021-12-24 上海寒武纪信息科技有限公司 Task processing method and device, computer equipment and storage medium

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102096687A (en) * 2009-12-14 2011-06-15 阿里巴巴集团控股有限公司 Method and platform for scheduling tasks
US20150149636A1 (en) * 2013-11-25 2015-05-28 International Business Machines Corporation Cross-platform workload processing
CN107870807A (en) * 2016-09-26 2018-04-03 平安科技(深圳)有限公司 A kind of cross-platform method for scheduling task and device

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107016479A (en) * 2016-01-28 2017-08-04 五八同城信息技术有限公司 Task scheduling and managing method, apparatus and system
CN107016480B (en) * 2016-01-28 2021-02-02 五八同城信息技术有限公司 Task scheduling method, device and system
CN107483649A (en) * 2017-10-12 2017-12-15 福建富士通信息软件有限公司 A kind of cross-platform log collection method based on Kaa services

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102096687A (en) * 2009-12-14 2011-06-15 阿里巴巴集团控股有限公司 Method and platform for scheduling tasks
US20150149636A1 (en) * 2013-11-25 2015-05-28 International Business Machines Corporation Cross-platform workload processing
CN107870807A (en) * 2016-09-26 2018-04-03 平安科技(深圳)有限公司 A kind of cross-platform method for scheduling task and device

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111984390A (en) * 2020-08-31 2020-11-24 平安医疗健康管理股份有限公司 Task scheduling method, device, equipment and storage medium

Also Published As

Publication number Publication date
CN108710532A (en) 2018-10-26
CN108710532B (en) 2023-05-30

Similar Documents

Publication Publication Date Title
WO2019223182A1 (en) Cross-scheduling platform dependency implementation method and apparatus, device, and storage medium
US11314698B2 (en) Dynamically performing data processing in a data pipeline system
US11232015B2 (en) Automated software verification service
US11347555B2 (en) Integrated system to distribute and execute complex applications
US8146060B2 (en) Data processing system and method for execution of a test routine in connection with an operating system
WO2019223173A1 (en) High-frequency task scheduling method and system, and computer device and storage medium
US8495615B2 (en) Method, system and computer program for distributing software patches
US11663025B2 (en) Maintenance of and caching of suspended virtual computers in a pool of suspended virtual computers
CN112668386A (en) Long running workflows for document processing using robotic process automation
US20150263900A1 (en) High performance distributed computing environment particularly suited for reservoir modeling and simulation
US8640121B2 (en) Facilitating multi-installer product installations
US20110320882A1 (en) Accelerated virtual environments deployment troubleshooting based on two level file system signature
US20020144249A1 (en) Method and apparatus for installing and upgrading an application in a computer system
JP2001147901A (en) Method and system for scheduling external job inside distributed processing system having local job control system
US11356508B1 (en) Retry strategies for handling failures during continuous delivery of software artifacts in a cloud platform
US20230334334A1 (en) Method and apparatus of executing dynamic graph for neural network computation
KR20150022849A (en) Auto-update while running client interface with handshake
WO2022121971A1 (en) Front-end project processing method and apparatus, device, management system, and storage medium
US20230019354A1 (en) System and method for the remote execution of one or more arbitrarily defined workflows
CN110019144A (en) A kind of method and system of big data platform data O&M
JP4109153B2 (en) Job control apparatus, job control file, job control method, and job control program
US20200225923A1 (en) Abort installation of firmware bundles
CN110109747B (en) Apache Spark-based data exchange method, system and server
US11321120B2 (en) Data backup method, electronic device and computer program product
Buchert et al. A workflow-inspired, modular and robust approach to experiments in distributed systems

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18919748

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18919748

Country of ref document: EP

Kind code of ref document: A1