CN113553152A

CN113553152A - Job scheduling method and device

Info

Publication number: CN113553152A
Application number: CN202110817908.1A
Authority: CN
Inventors: 王泽普; 张世瑛; 赵吉昆; 梁晔华
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2021-07-20
Filing date: 2021-07-20
Publication date: 2021-10-26

Abstract

The embodiment of the application provides a job scheduling method and a job scheduling device, which can be used in the technical field of big data, and the method comprises the following steps: a database corresponding to the periodic polling service cloud platform, if the database currently has the operation of the target application to be executed, acquiring the operation information of the target application and a corresponding operation script; and judging whether the number of the parallel jobs of the target application in the target cluster corresponding to the target application is smaller than a maximum job number threshold value or not, if not, executing the job of the target application in the target cluster corresponding to the service cloud platform based on the execution node corresponding to the target application according to the job information of the target application and the corresponding job script. According to the method and the device, the technical risk of job scheduling can be effectively reduced on the basis of improving the timeliness of job scheduling for the service cloud platform, the application safety and the stable operation of the service cloud platform are guaranteed, the technical threshold of job scheduling for the service cloud platform can be reduced, and the user experience of accessing the application to the service cloud platform by a user is effectively improved.

Description

Job scheduling method and device

Technical Field

The application relates to the technical field of data processing, in particular to the technical field of big data, and specifically relates to a job scheduling method and device.

Background

The big data service cloud platform of the enterprise provides services such as data access, storage, calculation, safety management and resource management for various proprietary big data analysis applications. With the continuous improvement of platform construction, the technical system is increasingly huge, the loaded business functions are increasingly abundant, large data are regarded as strategic resources by large enterprises while the platform operation and maintenance system is continuously expanded, and the application range of the Hadoop ecosystem is increasingly wide.

Scheduling is used as a management mode of an enterprise big data platform for running of each service, and conventional scheduling is offline batch processing, but due to the fact that the requirement for timeliness of the existing service is higher and higher, conventional batch processing cannot meet the requirement for accessing and using of each application, and more tasks such as real-time batch processing, real-time data analysis and mining and the like are executed on a cluster. Offline batch processing cannot meet timeliness requirements, but making the application directly connected to the cluster has technical risks and technical thresholds. That is to say, the existing job scheduling mode for the service cloud platform has the problems that the timeliness requirement and the safety requirement cannot be met at the same time, the technical threshold requirement cannot be reduced, and the like.

Disclosure of Invention

Aiming at the problems in the prior art, the application provides a job scheduling method and device, which can effectively reduce the technical risk of job scheduling on the basis of improving the timeliness of job scheduling for a service cloud platform, further ensure the application safety and stable operation of the service cloud platform, effectively reduce the technical threshold of job scheduling for the service cloud platform, and effectively improve the user experience of accessing applications into the service cloud platform by a user.

In order to solve the technical problem, the application provides the following technical scheme:

in a first aspect, the present application provides a job scheduling method, including:

a database corresponding to the periodic polling service cloud platform, if the database currently has the operation of the target application to be executed, acquiring the operation information of the target application and a corresponding operation script;

and judging whether the number of parallel jobs of the target application in a target cluster corresponding to the target application is smaller than a maximum job number threshold value or not, if not, executing the job of the target application in the target cluster corresponding to the service cloud platform based on an execution node corresponding to the target application according to job information of the target application and a corresponding job script.

Further, still include:

receiving a job scheduling request based on a preset application server, wherein the job scheduling request comprises: the service cloud platform comprises an identification of an application to be accessed to the service cloud platform, operation information and an operation script;

and taking the corresponding relation among the identification of the application to be accessed to the service cloud platform, the operation information and the operation script as the operation of a target application to be executed, and storing the operation of the target application to be executed into an operation scheduling table in a database corresponding to the service cloud platform.

Further, before the determining whether the number of jobs currently executed by the target application in the corresponding target cluster is smaller than the maximum job number threshold, the method further includes:

calling cluster information and cluster types to be accessed by the target application from a preset application login information table, wherein the application login information table is used for storing the identification of each application, the corresponding relation between the cluster information and the cluster types;

and respectively determining a target cluster corresponding to the target application and an execution node corresponding to the target cluster according to the cluster type.

and calling a maximum job number threshold value of the target application in the target cluster from a preset application job control table, wherein the application job control table is used for storing the corresponding relation among the identifier of each application, the cluster type and the maximum job number threshold value.

Further, still include:

and if the number of the parallel jobs of the target application in the corresponding target cluster is equal to the maximum job number threshold value, the jobs of the target application are stored in the database again, so that the job information of the target application and the corresponding job script are acquired when the database is polled next time.

Further, still include:

and sending a message for informing that the number of the parallel jobs of the target application in the corresponding target cluster reaches the maximum limit to a preset application server.

Further, still include:

and if the corresponding execution result data is generated after the operation of the target application is executed in the target cluster, exporting the execution result data to a preset application server.

In a second aspect, the present application provides a job scheduling apparatus, comprising:

the operation polling module is used for periodically polling a database corresponding to the service cloud platform, and acquiring operation information of a target application and a corresponding operation script if the database currently has an operation of the target application to be executed;

and the job execution module is used for judging whether the number of parallel jobs of the target application in the target cluster corresponding to the target application is smaller than a maximum job number threshold value or not, and if not, executing the job of the target application in the target cluster corresponding to the service cloud platform based on the execution node corresponding to the target application according to the job information of the target application and the corresponding job script.

In a third aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the job scheduling method when executing the program.

In a fourth aspect, the present application provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the job scheduling method.

According to the technical scheme, the job scheduling method and the job scheduling device provided by the application comprise the following steps: a database corresponding to the periodic polling service cloud platform, if the database currently has the operation of the target application to be executed, acquiring the operation information of the target application and a corresponding operation script; judging whether the number of the parallel jobs of the target application in the corresponding target cluster is smaller than the maximum job number threshold value or not, if not, executing the job of the target application in the target cluster corresponding to the service cloud platform according to the job information of the target application and the corresponding job script based on the execution node corresponding to the target application, by arranging the database for storing the application operation to be executed, quasi-real-time access, access and the like of each cluster in the service cloud platform can be realized, thereby effectively reducing the technical risk of job scheduling on the basis of improving the timeliness of job scheduling for the service cloud platform, the application safety and the operation stability of the service cloud platform can be further ensured, meanwhile, the technical threshold of operation scheduling on the service cloud platform can be effectively reduced, and the user experience of accessing the application to the service cloud platform by a user can be effectively improved; by judging whether the number of the parallel jobs of the target application in the corresponding target cluster is smaller than the maximum job number threshold value or not, the number of the threads of the same application accessing the service cloud platform in parallel can be effectively controlled, so that the operation stability and the application universality of the application access job calling on the service cloud platform can be further improved, and the user experience of the application access service cloud platform can be further improved.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a schematic diagram of a relationship between a job scheduling apparatus and an application server and a client device, respectively, in an embodiment of the present application.

Fig. 2 is a first flowchart illustrating a job scheduling method in an embodiment of the present application.

Fig. 3 is a second flowchart of a job scheduling method in the embodiment of the present application.

Fig. 4 is a third flowchart illustrating a job scheduling method in the embodiment of the present application.

Fig. 5 is a fourth flowchart illustrating a job scheduling method in the embodiment of the present application.

Fig. 6 is a fifth flowchart illustrating a job scheduling method in an embodiment of the present application.

Fig. 7 is a sixth flowchart illustrating a job scheduling method in an embodiment of the present application.

Fig. 8 is a seventh flowchart illustrating a job scheduling method in the embodiment of the present application.

Fig. 9 is a schematic structural diagram of a job scheduling apparatus in the embodiment of the present application.

FIG. 10 is a flowchart of a job scheduling method provided by an application example of the present application.

Fig. 11 is a schematic diagram of a logic architecture of an execution node provided in an application example of the present application.

Fig. 12 is a schematic structural diagram of an electronic device in an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

It should be noted that the job scheduling method and apparatus disclosed in the present application may be used in the field of big data technology, and may also be used in any field other than the field of big data technology.

Aiming at the problems that the existing job scheduling mode aiming at the service cloud platform cannot meet the timeliness requirement and the safety requirement and lower the technical threshold requirement and the like, the embodiment of the application provides a job scheduling method, and the database for storing the application job to be executed is arranged, so that the quasi-real-time access, access and the like of each cluster applied to the service cloud platform can be realized, the technical risk of job scheduling can be effectively reduced on the basis of improving the timeliness of the job scheduling aiming at the service cloud platform, the application safety and the running stability of the service cloud platform can be further ensured, the technical threshold of job scheduling on the service cloud platform can be effectively reduced, and the user experience that a user accesses the application to the service cloud platform is effectively improved; by judging whether the number of the parallel jobs of the target application in the corresponding target cluster is smaller than the maximum job number threshold value or not, the number of the threads of the same application accessing the service cloud platform in parallel can be effectively controlled, so that the operation stability and the application universality of the application access job calling on the service cloud platform can be further improved, and the user experience of the application access service cloud platform can be further improved.

Based on the above, the present application further provides a job scheduling apparatus for implementing the job scheduling method provided in one or more embodiments of the present application, where the job scheduling apparatus may be a server, see fig. 1, the job scheduling apparatus may be in communication connection with an application server and each client device in sequence by itself or through a third-party server, and the job scheduling apparatus may receive, through the application server, a job scheduling polling start instruction sent by the client device, and periodically poll a database corresponding to a service cloud platform according to the job scheduling start instruction, and if there is a job of a target application to be executed in the database, obtain job information of the target application and a corresponding job script; judging whether the number of parallel jobs of the target application in a target cluster corresponding to the target application is smaller than a maximum job number threshold value or not, if not, executing the job of the target application in the target cluster corresponding to the service cloud platform according to job information of the target application and a corresponding job script based on an execution node corresponding to the target application, and if corresponding execution result data is generated after the job of the target application is executed in the target cluster, the job scheduling device can export the execution result data to a preset application server so that the application server can send the execution result data to client equipment of a user.

In another practical application scenario, the aforementioned part of the job scheduling apparatus for performing job scheduling may be executed in the server as described above, or all operations may be completed in the user end device. Specifically, the selection may be performed according to the processing capability of the user end device, the limitation of the user usage scenario, and the like. This is not a limitation of the present application. If all the operations are completed in the customer premise equipment, the customer premise equipment may further include a processor for performing specific processing of job scheduling.

It is understood that the mobile terminal may include any mobile device capable of loading an application, such as a smart phone, a tablet electronic device, a network set-top box, a portable computer, a Personal Digital Assistant (PDA), a vehicle-mounted device, a smart wearable device, and the like. Wherein, intelligence wearing equipment can include intelligent glasses, intelligent wrist-watch, intelligent bracelet etc..

The mobile terminal may have a communication module (i.e., a communication unit), and may be communicatively connected to a remote server to implement data transmission with the server. The server may include a server on the task scheduling center side, and in other implementation scenarios, the server may also include a server on an intermediate platform, for example, a server on a third-party server platform that is communicatively linked to the task scheduling center server. The server may include a single computer device, or may include a server cluster formed by a plurality of servers, or a server structure of a distributed apparatus.

The server and the mobile terminal may communicate using any suitable network protocol, including network protocols not yet developed at the filing date of this application. The network protocol may include, for example, a TCP/IP protocol, a UDP/IP protocol, an HTTP protocol, an HTTPS protocol, or the like. Of course, the network Protocol may also include, for example, an RPC Protocol (Remote Procedure Call Protocol), a REST Protocol (Representational State Transfer Protocol), and the like used above the above Protocol.

The following embodiments and application examples are specifically and individually described in detail.

In order to solve the problems that the existing job scheduling method for the service cloud platform cannot meet the timeliness requirement and the security requirement and can reduce the technical threshold requirement at the same time, the present application provides an embodiment of a job scheduling method, and referring to fig. 2, the job scheduling method executed based on a job scheduling device specifically includes the following contents:

step 100: and periodically polling a database corresponding to the service cloud platform, and if the database currently has the operation of the target application to be executed, acquiring the operation information of the target application and a corresponding operation script.

It is understood that the interval time of the periodic polling may be set according to the actual application situation, and specifically may be between 0.1 second and 24 hours, preferably between 1 and 5 seconds, for example, polling the database of the service cloud platform every 3 seconds.

Step 200: judging whether the number of the parallel jobs of the target application in the corresponding target cluster is smaller than a maximum job number threshold value or not, if not, executing step 210;

step 210: and executing the operation of the target application in the target cluster corresponding to the service cloud platform according to the operation information of the target application and the corresponding operation script based on the execution node corresponding to the target application.

As can be seen from the above description, the job scheduling method provided in the embodiment of the present application, by setting the database for storing the application job to be executed, can implement quasi-real-time access, and the like of each cluster applied in the service cloud platform, and further can effectively reduce the technical risk of job scheduling on the basis of improving the timeliness of job scheduling for the service cloud platform, and further can ensure application security and stable operation of the service cloud platform, and at the same time, can effectively reduce the technical threshold of job scheduling for the service cloud platform, and effectively improve the user experience of accessing the application to the service cloud platform by a user; by judging whether the number of the parallel jobs of the target application in the corresponding target cluster is smaller than the maximum job number threshold value or not, the number of the threads of the same application accessing the service cloud platform in parallel can be effectively controlled, so that the operation stability and the application universality of the application access job calling on the service cloud platform can be further improved, and the user experience of the application access service cloud platform can be further improved.

In order to further improve the reliability and effectiveness of subsequent job scheduling, in an embodiment of the job scheduling method provided by the present application, referring to fig. 3, before step 100 of the job scheduling method, the following contents are specifically included:

step 010: receiving a job scheduling request based on a preset application server, wherein the job scheduling request comprises: and the identification, the operation information and the operation script of the application to be accessed into the service cloud platform.

Step 020: and taking the corresponding relation among the identification of the application to be accessed to the service cloud platform, the operation information and the operation script as the operation of a target application to be executed, and storing the operation of the target application to be executed into an operation scheduling table in a database corresponding to the service cloud platform.

It is understood that the

steps

010 and 020 can be executed before the step 100, or can be executed at any time, and as long as the job scheduling request is received, the job scheduling request can be stored in real time.

Specifically, a user can connect to a service URL (Uniform Resource Locator) through a URL or a domain name based on an application server, and send a job information record consisting of job information and a corresponding job script to a database such as a BDSP repository through an interface based on the service URL.

As can be seen from the above description, according to the job scheduling method provided in the embodiment of the present application, by uniformly storing information in the job scheduling request sent by the user in the database corresponding to the service cloud platform, an effective and reliable data basis can be provided for periodically polling the database corresponding to the service cloud platform, so that the reliability and effectiveness of subsequent job scheduling can be further improved, and the efficiency of job scheduling can be further improved.

In order to improve the efficiency and the automation degree of obtaining cluster information and cluster types to be accessed by a target application, referring to fig. 4, in an embodiment of a job scheduling method provided by the present application, the following contents are specifically included between step 100 and step 200 of the job scheduling method:

step 110: calling cluster information and cluster types to be accessed by the target application from a preset application login information table, wherein the application login information table is used for storing the identification of each application, the corresponding relation between the cluster information and the cluster types;

step 120: and respectively determining a target cluster corresponding to the target application and an execution node corresponding to the target cluster according to the cluster type.

Specifically, an application login information table may be set, where the fields may include:

"application name APP _ ID, access cluster information log _ INFO, access cluster TYPE log _ TYPE, and login authentication information";

the login authentication information, such as the authority authentication file of the user logging in Hive, may be stored in each server. And after the user operation is submitted, the execution node is connected with the database to read login authentication information.

As can be seen from the above description, the job scheduling method provided in the embodiment of the present application, by accessing the application login information table, can effectively improve the efficiency and the degree of automation for obtaining the cluster information and the cluster type to be accessed by the target application, and further can effectively improve the efficiency and the degree of automation for determining the target cluster and the execution node corresponding to the target application, so as to further improve the reliability and the effectiveness of subsequent job scheduling, and further improve the efficiency of job scheduling.

In order to improve the efficiency and the automation degree of obtaining the maximum job number threshold, referring to fig. 5, in an embodiment of the job scheduling method provided by the present application, the following contents are further specifically included between step 120 and step 200 of the job scheduling method:

step 130: and calling a maximum job number threshold value of the target application in the target cluster from a preset application job control table, wherein the application job control table is used for storing the corresponding relation among the identifier of each application, the cluster type and the maximum job number threshold value.

Specifically, after the user job is submitted, the execution node is connected with the database to read login authentication information, and simultaneously, the parallel job thread number of the user is read; if the number of parallel operation threads exceeds the maximum limit when the user submits the operation, the operation cannot be continuously submitted to the cluster, and when the node polling operation list is executed next time, the operation is submitted again after whether the number of the operation threads of the user is in line with the number of the operation threads of the user is checked again.

As can be seen from the above description, the job scheduling method provided in this embodiment of the present application, by setting the application job control table and calling the maximum job number threshold corresponding to the target application in the target cluster in the application job control table, can effectively improve the efficiency and the degree of automation for obtaining the maximum job number threshold, and further can effectively improve the efficiency and the degree of automation for determining whether the number of jobs currently performed by the target application in the corresponding target cluster is smaller than the maximum job number threshold, so as to further improve the reliability and the effectiveness of subsequent job scheduling, and further improve the efficiency of job scheduling.

In order to effectively control the number of threads accessing the service cloud platform in parallel by the same application, referring to fig. 6, in an embodiment of the job scheduling method provided by the present application, after step 200 of the job scheduling method, the following contents are further specifically included:

if the number of the parallel jobs of the target application in the corresponding target cluster is equal to the maximum job number threshold after the judgment of the step 200, executing a step 300;

step 300: and if the number of the parallel jobs of the target application in the corresponding target cluster is equal to the maximum job number threshold value, the jobs of the target application are stored in the database again, so that the job information of the target application and the corresponding job script are acquired when the database is polled next time.

It can be understood that, each time a job to be executed is received, the job scheduling apparatus performs the determination of the maximum job number threshold, and performs the subsequent job scheduling task only when the number of jobs currently executed by the target application in the target cluster corresponding to the target application is less than the maximum job number threshold, so that the situation that the number of jobs currently executed by the target application in the target cluster corresponding to the target application is greater than the maximum job number threshold does not occur, and therefore, in step 300, only the number of jobs currently executed by the target application in the target cluster corresponding to the target application is equal to the maximum job number threshold.

Specifically, the execution node obtains the minimum parallelizable thread number and the maximum parallelizable thread number of the application APP corresponding to the application identifier in a preset application job control table according to the application identifier in the received job information, judges whether the current parallelizable maximum thread number of the application corresponding to the job to be executed is equal to the searched maximum parallelizable thread number, if so, indicates that the current parallelizable job thread number of the application corresponding to the job submitted by the user exceeds the maximum limit, the job cannot be continuously submitted to the cluster, and when the next execution node polls the job table, the execution node again checks whether the user job thread number is in accordance with the job to be submitted; and if the current parallel maximum thread number of the application corresponding to the to-be-executed job is smaller than the searched parallel maximum thread number, executing the application job in the corresponding service cloud cluster according to the job information and the job script of the to-be-executed application job.

As can be seen from the above description, according to the job scheduling method provided in this embodiment of the present application, when the number of jobs currently running in parallel in the corresponding target cluster of the target application reaches the maximum limit, execution of the corresponding job scheduling content is rejected, and the job number is determined again in the next polling, so that the number of threads that the same application runs in parallel to access the service cloud platform can be effectively controlled, and further, the running stability and the applicability of job scheduling for application access on the service cloud platform can be further improved.

In order to further improve the user experience with the requirement of the application access service cloud platform, in an embodiment of the job scheduling method provided by the present application, referring to fig. 7, the following content is further specifically included after step 300 in the job scheduling method:

step 400: and sending a message for informing that the number of the parallel jobs of the target application in the corresponding target cluster reaches the maximum limit to a preset application server.

As can be seen from the above description, in the job scheduling method provided in this embodiment of the present application, a message for informing that the number of parallel jobs currently in the corresponding target cluster of the target application has reached the maximum limit is sent to the application server, so that the application server forwards the message to the client device of the user sending the job scheduling request, which can further improve user experience with a requirement of accessing the application to the service cloud platform, and can effectively improve efficiency and reliability of the user knowing the message that job scheduling is temporarily unavailable currently.

In order to enable a user to quickly and reliably obtain execution result data corresponding to a job of a target application, referring to fig. 8, an embodiment of a job scheduling method provided in the present application further includes the following steps after step 210:

step 500: and if the corresponding execution result data is generated after the operation of the target application is executed in the target cluster, exporting the execution result data to a preset application server.

Specifically, if the service cloud cluster returns the export data corresponding to the application job, the export data is sent to a special GTP node as an export job, so that the GTP node (the GTP node is an execution node where GTP is located) sends the export job to the application server, so that a user can directly obtain the export job.

As can be seen from the above description, in the job scheduling method provided in this embodiment of the present application, by exporting, to the corresponding application server, the execution result data generated after the job of the target application is executed in the target cluster, the application server can send the execution result data to the client device of the user, so that the user can quickly and reliably obtain the execution result data corresponding to the job of the target application, and user experience can be further improved.

In terms of software, in order to solve the problem that the existing job scheduling method for the service cloud platform cannot simultaneously satisfy the timeliness requirement, the security requirement, the technical threshold requirement reduction and the like, the present application provides an embodiment of a job scheduling apparatus for executing all or part of the contents in the job scheduling method, and referring to fig. 9, the job scheduling apparatus specifically includes the following contents:

the job polling module 10 is configured to periodically poll a database corresponding to the service cloud platform, and if a job of a target application to be executed currently exists in the database, obtain job information of the target application and a corresponding job script.

And the job execution module 20 is configured to determine whether the number of parallel jobs of the target application in the target cluster corresponding to the target application is smaller than a maximum job number threshold, and if not, execute the job of the target application in the target cluster corresponding to the service cloud platform according to the job information of the target application and the corresponding job script based on the execution node corresponding to the target application.

The embodiment of the job scheduling apparatus provided in the present application may be specifically configured to execute the processing procedure of the embodiment of the job scheduling method in the foregoing embodiment, and the functions of the embodiment are not described herein again, and refer to the detailed description of the embodiment of the method.

As can be seen from the above description, the job scheduling apparatus provided in the embodiment of the present application, by setting the database for storing the application job to be executed, can implement quasi-real-time access, and the like of each cluster applied in the service cloud platform, and further can effectively reduce the technical risk of job scheduling on the basis of improving the timeliness of job scheduling for the service cloud platform, and further can ensure application security and stable operation of the service cloud platform, and at the same time, can effectively reduce the technical threshold of job scheduling for the service cloud platform, and effectively improve the user experience of accessing the application to the service cloud platform by a user; by judging whether the number of the parallel jobs of the target application in the corresponding target cluster is smaller than the maximum job number threshold value or not, the number of the threads of the same application accessing the service cloud platform in parallel can be effectively controlled, so that the operation stability and the application universality of the application access job calling on the service cloud platform can be further improved, and the user experience of the application access service cloud platform can be further improved.

In order to further explain the scheme, the application example provides a job scheduling method, and relates to the technical field of large data platform quasi-real-time batch processing.

In the application example of the application, a user submits job information and an execution statement through a URL (uniform resource locator), a scheduling service records the job information and the execution statement in a database, a subsequent scheduling service polls a database job scheduling table every three seconds, if an executable job sends the job information to an execution node, the execution node is connected with a cluster according to the user information, submits a task and monitors the task state in real time, and the database is updated after the task is finished and a result file is transmitted to a corresponding server through GTP (general packet transport protocol).

The application example of the application provides a job scheduling method which mainly comprises the following components:

1. the database Oracle is used for storing user information and other contents. The following information table is designed:

configuring Config, flexibly scheduling a configuration information table by each platform application, and using the maximum thread number, the minimum thread number and the like;

the TASK scheduling TASK table is used for storing TASK execution records and inquiring information such as TASK states;

the SCRIPT TYPE SCRIPT _ TYPE refers to a detailed SCRIPT related to various interfaces.

The specific description may relate to user control, as follows:

an application login information table may be set, wherein the fields may include:

the login authentication information, such as the authority authentication file of the user logging in Hive, may be stored in each server.

After the user operation is submitted, the execution node is connected with the database to read login authentication information, and simultaneously, the number of parallel operation threads of the user is read;

if the number of parallel operation threads exceeds the maximum limit when the user submits the operation, the operation cannot be continuously submitted to the cluster, and when the node polling operation list is executed next time, the operation is submitted again after whether the number of the operation threads of the user is in line with the number of the operation threads of the user is checked again.

For example, the application login information table is shown in table 1:

TABLE 1

APP_ID	LOGON_INFO	LOGON_TYPE
			F-ROMA	etlroma/hadoop@HADOOP.COM…	1
F-MPVS2	mpvs_user……	1002

As shown in the application login information table: wherein the LOGON _ INFO field contents contain: the method comprises the steps of connecting an HD cluster through an application program interface JDBC and connecting necessary information contained in an MPP cluster through a standard graph query language, wherein LOGON _ TYPE is an access cluster TYPE, 1 of the LOGON _ TYPE identifies a hadoop cluster, and 1002 represents the MPP cluster.

The application job control table is shown in table 2:

TABLE 2

APP_ID	TASK_TYPE	MIN_THREADS	MAX_THREADS
				F-MPVS	1	1	2

As shown in the application job control table: the TYPE of the TASK _ TYPE field is consistent with the TYPE of the LOGON _ TYPE field in the application login information table, MIN _ THREADS represents the minimum thread number of the corresponding application APP which can be in parallel, MAX _ THREADS represents the maximum thread number of the corresponding application APP which can be in parallel, and the MAX _ THREADS is used for limiting the number of the jobs of the application which run simultaneously.

See table 3 for functional content of the schedule:

TABLE 3

It is to be understood that the job referred to in this application refers to a job or TASK, and the job table and the TASK table each refer to a job schedule table stored in the database.

2. Service composed of load balancing of multiple servers

The services provided by each node are consistent, and the services comprise Tomcat pool and Worker pool. Tomcat pool is used to capture the job information obtained by url, and Worker pool is used to execute the job.

Based on the above, referring to fig. 10, the job scheduling method provided by the present application specifically includes the following contents:

s1: the user connects to a service URL (Uniform Resource Locator) through a URL or a domain name based on the application server, and transmits job information records of job information and corresponding job scripts to a database such as a BDSP repository through an interface based on the service URL.

S2: the scheduling service writes the job information into a job scheduling table, namely a task table, in the database, wherein the job scheduling table is used for storing the corresponding relation between the job information of the application of the service cloud cluster to be accessed and the job script.

S3: scheduling services such as ASYNC service and the like can poll the task table periodically (for example, every 3 seconds), if a job to be executed is read in the task table, job information of an application to be executed is obtained and stored in a working pool, wherein the job information of the application can be inquired from a preset application login information table, that is, information of a service cloud cluster applying for access and an access cluster category corresponding to an application identifier (for example, an application name) are included.

S4: according to the access cluster type in the application job information, determining respective corresponding execution nodes of the jobs to be executed in the current job pool respectively, that is, allocating the jobs to the corresponding type execution nodes for execution, for example, the jobs of the HADOOP type may be executed by HD task nodes, and the jobs of the MPP type may be executed in MPP task nodes (MPP is hua ge gausdb database, which is different from the HADOOP architecture).

S5: the execution node obtains the minimum parallelizable thread number and the maximum parallelizable thread number of the application APP corresponding to the application identifier from a preset application operation control table according to the application identifier in the received operation information, judges whether the current parallelizable maximum thread number of the application corresponding to the operation to be executed is equal to the searched maximum parallelizable thread number, if so, indicates that the current parallelizable operation thread number corresponding to the operation submitted by the user exceeds the maximum limit, the operation cannot be continuously submitted to the cluster, and when the next execution node polls the operation table, the execution node again checks whether the operation thread number of the user accords with the operation to submit the operation; and if the current parallel maximum thread number of the application corresponding to the to-be-executed job is smaller than the searched parallel maximum thread number, executing the application job in the corresponding service cloud cluster according to the job information and the job script of the to-be-executed application job.

S6: and if the service cloud cluster returns the export data corresponding to the application job, sending the export data serving as the export job to a special GTP node, so that the GTP node (the GTP node refers to an execution node where GTP is located) sends the export job to the application server, and a user can directly obtain the export job.

The flexible scheduling function can be formed by load balancing of a plurality of execution nodes, and the domain name can be uniformly used for providing service for the outside. The execution node is composed of Tomcat pool and Worker pool. The Tomcat pool is used to capture the job information transmitted through url and return the interface access result to the application side. The Worker pool has a function of executing the operation, after polling the job executable by the job table in the database every three seconds, the job executable by the job table is marked and then put into the Worker pool of the Worker pool, then the cluster is connected according to the user connection information of the job in the LOGON _ INF table in the database, and the job is submitted through a public script to monitor the execution state of the job. Referring to fig. 11, the logical architecture of the execution node comprises the following steps:

1. submitting a job: the application side submits the operation and the script (operation program) to a service URL, and the service URL can adopt a server which is flexibly scheduled and deployed;

2. writing operation: flexibly scheduling and writing the captured job information into a database;

3. polling pull operation: worker pool will poll the database for executable jobs;

4. execute the Hive type job: the Worker pool executes the corresponding operation;

5. generating GTP operation and writing: generating export jobs to write to the database;

6. polling pull operation: a GTP node (the GTP node refers to an execution node where GTP is located) polls the export operation executable by the database and pulls the export operation into a work pool of the GTP node;

7. and (3) executing the operation: executing the export operation;

8. generating a data file: exporting the data file to the local;

9. and transmitting to the application server through GTP.

Based on the above, the job scheduling method provided by the application example of the present application includes the following advantages:

(1) quasi-real-time batch processing operation efficiency improvement based on big data service cloud platform

Through the flexible scheduling function, the user can realize second-level triggering from job submission to triggering, and the scheduling efficiency of batch processing jobs is greatly improved.

(2) High flexibility, single and batch analysis support and continuous and perfect scheduling function

Flexible scheduling enables single or multiple batches to be scheduled simultaneously, while also supporting the scheduling of other distributed databases. The flexible scheduling function has the functions of syntax checking, flexible scheduling job searching and killing interfaces and the like, and the service capability is further improved.

(3) The development threshold is low, and the cost and the risk of accessing the cluster are reduced

The flexible scheduling can realize the low threshold access cluster, the user only concerns about script writing, the flexible scheduling manages the user information and authority management, and the technical risk of directly connecting the cluster is avoided.

In terms of hardware, in order to solve the problems that an existing job scheduling method for a service cloud platform cannot meet timeliness requirements, safety requirements, technical threshold requirements reduction and the like at the same time, the present application provides an embodiment of an electronic device for implementing all or part of the contents in the job scheduling method, where the electronic device specifically includes the following contents:

fig. 12 is a schematic block diagram of a system configuration of an electronic device 9600 according to an embodiment of the present application. As shown in fig. 12, the electronic device 9600 can include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. Notably, this fig. 12 is exemplary; other types of structures may also be used in addition to or in place of the structure to implement telecommunications or other functions.

In one embodiment, the job scheduling functionality may be integrated into a central processor. Wherein the central processor may be configured to control:

As can be seen from the above description, according to the electronic device provided in the embodiment of the present application, by setting the database for storing the application job to be executed, quasi-real-time access, and the like of each cluster applied in the service cloud platform can be realized, so that on the basis of improving the timeliness of job scheduling for the service cloud platform, the technical risk of job scheduling can be effectively reduced, the application security and the operation stability of the service cloud platform can be further ensured, meanwhile, the technical threshold of job scheduling for the service cloud platform can be effectively reduced, and the user experience of accessing the application to the service cloud platform by a user can be effectively improved; by judging whether the number of the parallel jobs of the target application in the corresponding target cluster is smaller than the maximum job number threshold value or not, the number of the threads of the same application accessing the service cloud platform in parallel can be effectively controlled, so that the operation stability and the application universality of the application access job calling on the service cloud platform can be further improved, and the user experience of the application access service cloud platform can be further improved.

In another embodiment, the job scheduling apparatus may be configured separately from the central processor 9100, for example, the job scheduling apparatus may be configured as a chip connected to the central processor 9100, and the job scheduling function is realized by the control of the central processor.

As shown in fig. 12, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is noted that the electronic device 9600 also does not necessarily include all of the components shown in fig. 12; further, the electronic device 9600 may further include components not shown in fig. 12, which can be referred to in the related art.

As shown in fig. 12, a central processor 9100, sometimes referred to as a controller or operational control, can include a microprocessor or other processor device and/or logic device, which central processor 9100 receives input and controls the operation of the various components of the electronic device 9600.

The memory 9140 can be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. The information relating to the failure may be stored, and a program for executing the information may be stored. And the central processing unit 9100 can execute the program stored in the memory 9140 to realize information storage or processing, or the like.

The input unit 9120 provides input to the central processor 9100. The input unit 9120 is, for example, a key or a touch input device. Power supply 9170 is used to provide power to electronic device 9600. The display 9160 is used for displaying display objects such as images and characters. The display may be, for example, an LCD display, but is not limited thereto.

The memory 9140 can be a solid state memory, e.g., Read Only Memory (ROM), Random Access Memory (RAM), a SIM card, or the like. There may also be a memory that holds information even when power is off, can be selectively erased, and is provided with more data, an example of which is sometimes called an EPROM or the like. The memory 9140 could also be some other type of device. Memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application/function storage portion 9142, the application/function storage portion 9142 being used for storing application programs and function programs or for executing a flow of operations of the electronic device 9600 by the central processor 9100.

The memory 9140 can also include a data store 9143, the data store 9143 being used to store data, such as contacts, digital data, pictures, sounds, and/or any other data used by an electronic device. The driver storage portion 9144 of the memory 9140 may include various drivers for the electronic device for communication functions and/or for performing other functions of the electronic device (e.g., messaging applications, contact book applications, etc.).

The communication module 9110 is a transmitter/receiver 9110 that transmits and receives signals via an antenna 9111. The communication module (transmitter/receiver) 9110 is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as in the case of a conventional mobile communication terminal.

Based on different communication technologies, a plurality of communication modules 9110, such as a cellular network module, a bluetooth module, and/or a wireless local area network module, may be provided in the same electronic device. The communication module (transmitter/receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby implementing ordinary telecommunications functions. The audio processor 9130 may include any suitable buffers, decoders, amplifiers and so forth. In addition, the audio processor 9130 is also coupled to the central processor 9100, thereby enabling recording locally through the microphone 9132 and enabling locally stored sounds to be played through the speaker 9131.

An embodiment of the present application further provides a computer-readable storage medium capable of implementing all the steps in the job scheduling method in the foregoing embodiment, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements all the steps of the job scheduling method in which an execution subject is a server or a client in the foregoing embodiment, for example, when the processor executes the computer program, the processor implements the following steps:

As can be seen from the above description, the computer-readable storage medium provided in the embodiment of the present application, through setting the database for storing the application job to be executed, can implement quasi-real-time access, and the like of each cluster applied in the service cloud platform, and further can effectively reduce the technical risk of job scheduling on the basis of improving the timeliness of job scheduling for the service cloud platform, so as to ensure the application security and stable operation of the service cloud platform, and at the same time, can effectively reduce the technical threshold of job scheduling for the service cloud platform, and effectively improve the user experience of accessing the application to the service cloud platform by the user; by judging whether the number of the parallel jobs of the target application in the corresponding target cluster is smaller than the maximum job number threshold value or not, the number of the threads of the same application accessing the service cloud platform in parallel can be effectively controlled, so that the operation stability and the application universality of the application access job calling on the service cloud platform can be further improved, and the user experience of the application access service cloud platform can be further improved.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The principle and the implementation mode of the invention are explained by applying specific embodiments in the invention, and the description of the embodiments is only used for helping to understand the method and the core idea of the invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.

Claims

1. A job scheduling method, comprising:

2. The job scheduling method according to claim 1, further comprising:

3. The job scheduling method according to claim 1, before the determining whether the number of jobs currently being performed by the target application in parallel in the target cluster corresponding to the target application is less than a maximum job number threshold, further comprising:

4. The job scheduling method according to claim 1, before the determining whether the number of jobs currently being performed by the target application in parallel in the target cluster corresponding to the target application is less than a maximum job number threshold, further comprising:

5. The job scheduling method according to claim 1, further comprising:

6. The job scheduling method according to claim 5, further comprising:

7. The job scheduling method according to any one of claims 1 to 4, further comprising:

8. A job scheduling apparatus comprising:

9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the job scheduling method of any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium on which a computer program is stored, the computer program, when being executed by a processor, implementing the job scheduling method according to any one of claims 1 to 7.