CN113918525A

CN113918525A - Data exchange scheduling method, system, electronic device, medium, and program product

Info

Publication number: CN113918525A
Application number: CN202111279303.8A
Authority: CN
Inventors: 张雯; 丁文定; 徐平; 王金余
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2021-10-29
Filing date: 2021-10-29
Publication date: 2022-01-11

Abstract

The present disclosure provides a data exchange scheduling method, which can be used in the financial field or other fields, including: performing definition preprocessing on a file to be processed to generate job definition information and job dependency information of the file to be processed; according to the job definition information, the to-be-processed file is instantiated to obtain a job instance of the to-be-processed file, and the dependency instance information of the to-be-processed job is generated according to the job dependency information and the job instance; and judging whether the job to be processed meets the dependency of the front job or not according to the dependency example information, and if so, setting the job to be processed as the job to be scheduled and performing scheduling processing. The present disclosure also provides a data exchange scheduling system, an electronic device, a computer-readable storage medium, and a computer program product.

Description

Data exchange scheduling method, system, electronic device, medium, and program product

Technical Field

The present disclosure relates to the field of data processing technologies, and in particular, to a data exchange scheduling method, system, electronic device, computer-readable storage medium, and computer program product.

Background

With the continuous expansion of the scale of enterprise business systems, the data exchange platform helps the rapid and efficient data sharing among the systems in the enterprise, reduces the data processing links and improves the data processing efficiency. For an enterprise-level data exchange platform taking files as main objects, massive files need to be exchanged every day, the number, content, time and frequency of files downloaded by a file supplier (called an upstream for short) every day are different, each file relates to a plurality of demanders (called a downstream for short), the files of each demander need different processing operations for processing, and the execution sequence requirements among various operations are met. And matching the corresponding processing operation and operation execution link for each file reaching the platform by data exchange scheduling, and then performing operation scheduling processing.

At present, a processing mode of a data exchange job scheduling system is to use a mode of generating jobs and job dependencies in advance and triggering a scheduling system to schedule related jobs by detecting arrival of files, and this mode can solve and reduce human intervention in the scheduling system, but the prior art has at least two problems: 1) the operation generated in advance by the operation cannot adapt to the actually changed exchange data, because the system relates to more upstream, the subsequent operation cannot be started due to the fact that the file name and the definition of one upstream download are different, and if the content and the frequency of the upstream download change, the system cannot process the subsequent operation. 2) Generating full jobs ahead of time will occupy a lot of system resources and should be available for system service.

Disclosure of Invention

In order to solve the problems in the prior art, embodiments of the present disclosure provide a data exchange scheduling method, system, electronic device, computer-readable storage medium, and computer program product, so as to implement conversion and scheduling from a file to be processed to a job to be processed.

A first aspect of the present disclosure provides a data exchange scheduling method, including: performing definition preprocessing on a file to be processed to generate job definition information and job dependency information of the file to be processed; according to the job definition information, the to-be-processed file is instantiated to obtain a job instance of the to-be-processed file, and the dependency instance information of the to-be-processed job is generated according to the job dependency information and the job instance; and judging whether the job to be processed meets the dependency of the front job or not according to the dependency example information, and if so, setting the job to be processed as the job to be scheduled and performing scheduling processing.

Further, the defining preprocessing is performed on the file to be processed, and the job definition information and the job dependency information of the file to be processed are generated, including: acquiring basic information of a plurality of files to be processed; carrying out exchange requirement definition on the basic information of a plurality of files to be processed to generate exchange information; reading exchange information in preset time, and extracting a keyword field of the exchange information to generate a data definition file of a file to be processed; and generating job definition information according to the processing mode and the job type of each data record in the exchange information, and generating job dependency information according to the execution sequence of each data record and the job type in the exchange information.

Further, before the document to be processed is instantiated according to the job definition information, the method further includes: adding the downloaded directory information in the data definition file into a detection directory table; detecting whether list verification files are generated or not by detecting the directory information in the detection directory table; if yes, detecting whether the list check file is matched with the data definition file; if the data are matched, adding the data definition file to the processing directory file, and inputting the detected actual file information into the source data information file; and if not, adding the data definition file to the abnormal processing directory file.

Further, according to the job definition information, the to-be-processed file is instantiated to obtain a job instance of the to-be-processed file, which includes: and matching the file to be processed recorded in the source data information file with the job definition information and the data definition file, and converting the replacement macro into an example job to obtain a job example of the file to be processed.

Further, generating dependency instance information of the job to be processed according to the job dependency information and the job instance, including: and generating the dependency instance information of the job to be processed according to the job name in the job instance file and the post job name in the job dependency information.

Further, the judging whether the job to be processed meets the pre-job dependency includes: judging whether the job to be processed has a pre-job dependency or not; if not, setting the job to be processed as a job to be scheduled; if yes, judging whether the front operation is processed or not, and if the front operation is processed, setting the operation to be processed as the operation to be scheduled.

Further, the method further comprises: judging whether the job to be scheduled is a dead job, wherein the dead job represents the job to be scheduled which is not subjected to job processing within a preset time; if so, resetting the dead operation as the operation to be scheduled.

Further, setting the job to be processed as a job to be scheduled and performing scheduling processing, including: adding the job to be scheduled into a distribution job queue; sequentially taking out a plurality of jobs to be scheduled from the distribution job queue, distributing the jobs to be scheduled to idle node addresses in a job client, and setting the state of the jobs to be scheduled to be processed; and submitting the plurality of jobs to be scheduled to the job client for job processing, and updating the job state after the jobs to be processed are processed.

Further, the job status after the job to be processed is processed at least includes: in process or process completion or process failure.

A second aspect of the present disclosure provides a data exchange scheduling system, including: the file preprocessing module is used for performing definition preprocessing on the file to be processed and generating job definition information and job dependency information of the file to be processed; the job instantiation module is used for instantiating the file to be processed according to the job definition information to obtain a job instance of the file to be processed and generating dependency instance information of the job to be processed according to the job dependency information and the job instance; and the job scheduling module is used for judging whether the job to be processed meets the dependency of the front job or not according to the dependency example information, and if so, setting the job to be processed as the job to be scheduled and performing scheduling processing.

Further, the file preprocessing module is configured to perform definition preprocessing on the file to be processed, and generate job definition information and job dependency information of the file to be processed, and includes: acquiring basic information of a plurality of files to be processed; carrying out exchange requirement definition on the basic information of a plurality of files to be processed to generate exchange information; reading exchange information in preset time, and extracting a keyword field of the exchange information to generate a data definition file of a file to be processed; and generating job definition information according to the processing mode and the job type of each data record in the exchange information, and generating job dependency information according to the execution sequence of each data record and the job type in the exchange information.

Further, the system further comprises: the file inspection module is used for inspecting the file to be processed and comprises: adding the downloaded directory information in the data definition file into a detection directory table; detecting whether list check files are generated or not by detecting directory information in a detection directory table; if yes, detecting whether the list check file is matched with the data definition file; if the data are matched, adding the data definition file to the processing directory file, and inputting the detected actual file information into the source data information file; and if not, adding the data definition file to the abnormal processing directory file.

A third aspect of the present disclosure provides an electronic device, comprising: the data exchange scheduling method provided by the first aspect of the present disclosure is implemented by a memory, a processor and a computer program stored in the memory and executable on the processor.

A fourth aspect of the present disclosure provides a computer-readable storage medium on which a computer program is stored, which, when executed by a processor, implements the data exchange scheduling method provided by the first aspect of the present disclosure.

A fifth aspect of the present disclosure provides a computer program product comprising a computer program which, when executed by a processor, implements the data exchange scheduling method provided by the first aspect of the present disclosure.

The method and the system are divided into two stages of operation definition pre-generation and operation definition quick generation example operation, files needing to be processed are registered for multiple times at one time, the use cost of users is low, and the method and the system can dynamically generate operation according to actual file downloading of the users without limitation on file downloading time points, frequency and the like.

Drawings

For a more complete understanding of the present disclosure and the advantages thereof, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which:

fig. 1 schematically illustrates an application scenario of a data exchange scheduling method according to an embodiment of the present disclosure;

FIG. 2 schematically illustrates a flow diagram of a data exchange scheduling method according to an embodiment of the present disclosure;

FIG. 3 schematically illustrates a flow diagram of a definition pre-processing of a document to be processed according to an embodiment of the present disclosure;

FIG. 4 schematically illustrates a flow diagram of data file verification according to an embodiment of the present disclosure;

FIG. 5 schematically illustrates a flow diagram for converting a pending job to be scheduled, according to an embodiment of the present disclosure;

FIG. 6 schematically illustrates a flow diagram for scheduling a job to be scheduled, according to an embodiment of the present disclosure;

FIG. 7 schematically illustrates a block diagram of a data exchange scheduling system according to an embodiment of the present disclosure;

FIG. 8 schematically illustrates a block diagram of a data exchange scheduling system according to another embodiment of the present disclosure;

fig. 9 schematically shows a block diagram of an electronic device adapted to implement the above described method according to an embodiment of the present disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.

Where a convention analogous to "at least one of A, B and C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B and C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). Where a convention analogous to "A, B or at least one of C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B or C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).

Some block diagrams and/or flow diagrams are shown in the figures. It will be understood that some blocks of the block diagrams and/or flowchart illustrations, or combinations thereof, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the instructions, which execute via the processor, create means for implementing the functions/acts specified in the block diagrams and/or flowchart block or blocks. The techniques of this disclosure may be implemented in hardware and/or software (including firmware, microcode, etc.). In addition, the techniques of this disclosure may take the form of a computer program product on a computer-readable storage medium having instructions stored thereon for use by or in connection with an instruction execution system.

The embodiment of the disclosure provides a data exchange scheduling method, which can be applied to a data exchange platform, and avoids the problem of possible processing interruption or failure in processing through a dynamic job derivation mode when the number, content, time and frequency of massive files change. The construction of a file exchange path can be completed by simply defining the upstream and the downstream of the data, the file can be defined for multiple times, the file can be operated according to the actual upstream transmission condition, the generation of the file operation cannot be influenced by the change of the number, the content, the time and the frequency of the file, and the fault tolerance rate of the system is improved.

Fig. 1 schematically illustrates an exemplary system architecture 100 that may be applied to a data exchange scheduling method according to an embodiment of the present disclosure. It should be noted that fig. 1 is only an example of a system architecture to which the embodiments of the present disclosure may be applied to help those skilled in the art understand the technical content of the present disclosure, and does not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments or scenarios.

As shown in fig. 1, the system architecture 100 according to this embodiment may include

terminal devices

101, 102, 103, a network 104 and a server 105. The network 104 serves as a medium for providing communication links between the

terminal devices

101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

A user (e.g., a developer) may use the

terminal devices

101, 102, 103 to interact with the server 105 over the network 104 to receive or send messages, etc. The

terminal devices

101, 102, 103 may have installed thereon various communication client applications, such as various language software programming systems, testing systems, web browser applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).

The

terminal devices

101, 102, 103 may be various electronic devices having display screens and supporting web browsing, and the

terminal devices

101, 102, 103 provide an operating platform for upstream users, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The server 105 may be a server providing various services, which may be a deployment server of the method, such as a background management server (for example only) providing support for applications that users follow with the

terminal devices

101, 102, 103. The background management server may perform file processing on the received user request and the like, and feed back a processing result (e.g., job scheduling, processing, and the like) to the terminal device.

It should be noted that the data exchange scheduling method provided by the embodiment of the present disclosure may be generally executed by the server 105. Accordingly, the data exchange scheduling system provided by the embodiments of the present disclosure may be generally deployed in the server 105. The data exchange scheduling method provided by the embodiments of the present disclosure may also be performed by a server or a server cluster that is different from the server 105 and is capable of communicating with the

terminal devices

101, 102, 103 and/or the server 105. The server cluster may be a server cluster of a data exchange platform, and specifically may include: the data exchange platform server cluster carries out data pre-definition processing, instance data entry, job instantiation, job scheduling and processing processes. Accordingly, the scheduling system for data exchange provided by the embodiment of the present disclosure may also be disposed in a server or a server cluster different from the server 105 and capable of communicating with the

terminal devices

101, 102, 103 and/or the server 105.

It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

Fig. 2 schematically shows a flow chart of a data exchange scheduling method according to an embodiment of the present disclosure. As shown in fig. 2, the method includes: steps S201 to S203.

In operation S201, a definition preprocessing is performed on the file to be processed, and job definition information and job dependency information of the file to be processed are generated.

In operation S202, the to-be-processed file is instantiated according to the job definition information to obtain a job instance of the to-be-processed file, and dependent instance information of the to-be-processed job is generated according to the job dependent information and the job instance.

In operation S203, it is determined whether the job to be processed satisfies the pre-job dependency according to the dependency instance information, and if so, the job to be processed is set as a job to be scheduled and is scheduled.

An exemplary flow of each step of the data exchange scheduling method of the present embodiment is described in detail below.

In the embodiment of the present disclosure, the data exchange scheduling method may be applied to an electronic device, and the electronic device may include, but is not limited to, a server cluster, and the like. Each application system, a data exchange scheduling system and the like can be deployed in the electronic equipment, and all files to be downloaded which are input upstream are stored in the server. The file to be processed may be a file in any format, including but not limited to an xls format file.

According to an embodiment of the present disclosure, as shown in fig. 3, performing definition preprocessing on a file to be processed to generate job definition information and job dependency information of the file to be processed includes: steps S301 to S304.

In operation S301, basic information of a plurality of files to be processed is acquired.

In operation S302, the basic information of the plurality of files to be processed is defined by the exchange requirement, and exchange information is generated.

Specifically, the basic information of all the files to be downloaded recorded in the data exchange platform at the upstream is stored in the database of the data exchange platform, and includes, but is not limited to, a file name, a file field name (including an encoding type of the field), file downloading information, and the like. The input of each file can be used for all downstream, and the downstream selects the files in the system according to the needs, defines the needed processing mode and receives the catalog to complete the exchange requirement definition, and forms the exchange information. The file name representation distinguishes the only main key of the file to be processed, the file field name (including the coding type of the field) refers to the file content structure information, and the file downloading information refers to the directory of the upstream downloading data file.

In operation S303, the exchange information within the preset time is read, and the keyword field of the exchange information is extracted to generate the data definition file of the file to be processed.

In the embodiment of the disclosure, the step may be processed by a data preprocessor, and the data preprocessor reads the exchange information within a preset time and extracts the keyword field of the exchange information to generate a data definition file of the file to be processed. The preset time can be any time period set according to requirements, for example, 1 hour, 2 hours, 6 hours, 24 hours, etc., the keyword fields of the exchange information include, but are not limited to, file name, upstream identifier, downstream identifier, upstream download directory, downstream receiving directory, fields under processing mode, etc., and the data definition file generated thereby also includes, but is not limited to, file name, upstream identifier, downstream identifier, upstream download directory, downstream receiving directory, and contents under processing mode, etc.

In operation S304, job definition information is generated according to the processing method and the job type of each data record in the exchange information, and job dependency information is generated according to the execution sequence of each data record and the job type in the exchange information.

In the embodiment of the present disclosure, step S304 is performed by the data preprocessor, specifically, the corresponding handler command and job type are searched according to the processing mode of each data record, and job command definition splicing is performed, where the format of the job command definition splicing may be "processing command + # file source path # + # service date #" (# XXX # is a macro definition, and will be replaced by valid data in an instantiation stage), so as to generate job definition information, and the job definition information may be stored in a job definition file in a preset format.

Further, according to each data record in the exchange information and the execution sequence of the job type, generating job dependent information by using each job name and the corresponding post-job name relation record, wherein the job dependent information can be stored in a job dependent definition file in a preset format.

It should be noted that the file formats (e.g., data definition file, job dependent definition file, etc.) indicated above can be set according to actual requirements, and include, but are not limited to, xls format file, etc.

According to the embodiment of the present disclosure, after the definition preprocessing is performed on the file to be processed, and before the instantiation processing is performed on the file to be processed according to the job definition information, the method further includes a file verification process, where the file verification process may be performed by a data detector, and specifically includes: as shown in fig. 4, the download directory information in the data definition file is added to the detection directory table; detecting whether a list check file (namely, a CHK file) is generated or not by detecting directory information in a detection directory table; if yes, detecting whether the list check file is aligned, and then detecting whether the list check file is matched with the data definition file; and if the data are matched, adding the data definition file into the processing directory file, and inputting the detected actual file information into the source data information file. And if not, adding the data definition file to the abnormal processing directory file, and recording the error file into the data abnormal file. Specifically, detecting whether the list check file is in order to check whether the file exists in the current directory and is consistent in size, and inputting the detected actual file information into a source data information file, wherein the source data information file at least comprises contents such as a file name, a file processing directory, a processing type, a service date, an upstream name, a downstream name, a serial number, a job derivation state and the like. The file processing directory refers to a directory where a file is currently located; the processing type refers to a mode that a file needs to be processed, such as 1, 2, 3, and the like, and the numbers 1, 2, 3 represent different processing types, such as 1 can represent host decoding and the like, which can be set according to actual requirements; the upstream name and the downstream name correspond to the upstream identifier and the downstream identifier respectively; the sequence number refers to the sequence for identifying the downloading of the same file for multiple times on the same service date, so that the problem of uncertain upstream downloading frequency is solved; job-derived state includes, but is not limited to, pending or derived (schedule).

In the embodiment of the disclosure, a file verification process is set before file instantiation processing is performed, so that the problem that a job generated in advance by the job cannot adapt to actually changed exchange data is avoided, and when the system relates to more upstream, if the name and definition of a file downloaded from the upstream are different, the subsequent job cannot be called up, so that the subsequent steps cannot be normally realized.

In the embodiment of the present disclosure, the process of instantiating the to-be-processed file is completed by the job generator, specifically, instantiating the to-be-processed file according to the job definition information to obtain the job instance of the to-be-processed file includes: and matching the file to be processed recorded in the source data information file with the job definition information and the data definition file, and converting the replacement macro into an example job to obtain a job example of the file to be processed.

Specifically, a label is added to a record to be processed in a polling source data information file and the record is inserted into a source data temporary file, the source data temporary file searches for a corresponding job in a job definition file according to a file name and upper and lower cursor identifiers, a # file source path # in the job definition file is replaced by a file processing directory in the source data information file, a # service date # is replaced by a corresponding service log to form an effective job command, instantiated job information is set to be in a state to be verified, and a job instance of the file to be processed is obtained. The method comprises the steps that a job instance represents a file to be processed to conduct job instantiation, a corresponding job instance file is generated, and information related to the job instance is stored in the job instance file.

Further, generating dependency instance information of the job to be processed according to the job dependency information and the job instance, specifically including: and generating the dependency instance information of the job to be processed according to the job name in the job instance file and the post job name in the job dependency information. Dependency instance information, which may be stored in a dependency instance file, includes, but is not limited to, a pre-job name, a pre-job business date, a pre-job serial number, a post-job name, a post-job business date, a post-job serial number, whether dependencies have been satisfied, and the like.

In the embodiment of the present disclosure, the scheduling process of the job to be processed is performed by the scheduling server, as shown in fig. 5, specifically: judging whether the job to be processed meets the requirement of the front job, and specifically comprising the following steps: judging whether the job to be processed has a pre-job dependency or not; if not, setting the job to be processed as a job to be scheduled; if yes, judging whether the front operation is processed or not, and if the front operation is processed, setting the operation to be processed as the operation to be scheduled.

Specifically, to prevent the job to be scheduled from being unsuccessfully processed after a certain time, the method provided by the present disclosure further includes: judging whether the job to be scheduled is a dead job, wherein the dead job represents the job to be scheduled which is not subjected to job processing within a preset time; if so, resetting the dead operation as the operation to be scheduled. For example, it is determined whether there is a dead job which is not processed in a job for more than 10 minutes in the job to be scheduled, that is, a job which is not processed in a job processing or fails in a job, and such a job is reset to a job to be scheduled, and the scheduling process is performed again.

As shown in fig. 6, in the embodiment of the present disclosure, the scheduling processing on the job to be scheduled specifically includes: adding the job to be scheduled into a distribution job queue; sequentially taking out a plurality of jobs to be scheduled from the distribution job queue, distributing the jobs to be scheduled to idle node addresses in a job client, and setting the state of the jobs to be scheduled to be processed; and submitting the plurality of jobs to be scheduled to the job client for job processing, and updating the job state after the jobs to be processed are processed. Specifically, before distributing a plurality of jobs to be scheduled to job clients, the available state of the job clients needs to be checked regularly according to a registered client resource table, the base table is updated, available job client addresses and available resource numbers are obtained according to the base table to form a job client resource list, and the available job clients are selected according to the job client resource list to perform job scheduling processing. The number of the jobs to be scheduled, which are allocated to the available job client, is set according to data of the idle node addresses in the job client, which is not limited in this embodiment of the present disclosure.

And after the job client receives the distributed job to be scheduled, executing a job processing command, setting the state of the job in the base table to be processed after each job to be processed is processed, and adding '1' to the idle number of the resources of the job client. Meanwhile, the job client triggers dependent job scheduling, matches the finished job name with the job dependent information file to check whether subsequent jobs exist, and sets the dependent state of the post job to be satisfied if the subsequent jobs exist. The job status after the job to be processed is processed at least includes processing, completion of processing, failure of processing, and the like.

It should be noted that the preset time of 10 minutes and the job status after the job to be processed is processed in the foregoing embodiment are only exemplary illustrations and do not limit the embodiment of the present disclosure, and in the actual application process, the job which is not processed or fails to be processed in the preset time may be a job to be scheduled within 20 minutes or half an hour, and the job status may also be set according to actual requirements.

The following describes in detail the process of performing job definition and instantiation processing on a file to be processed according to an embodiment of the present disclosure with reference to a specific embodiment. The following description is only for assisting a person skilled in the art to understand the technical content of the present disclosure, and does not mean the intermediate information generated by each process of the embodiments of the present disclosure and the limitation of the applicable scenario.

Firstly, the upstream inputs basic information of all downloaded files in the system, and the specific content is as follows:

table 1 basic information of download file

In table 1, the file structure information file at least includes information such as a file field name, a field type, a field length, a decoding type, a decoded length, and a decoded position. And the downstream selects the files in the system according to the requirement and defines the required processing mode and the receiving catalog to complete the exchange requirement definition according to all the files input at the upstream so as to form the exchange information.

The data preprocessor reads the exchange information registered and generated in the last 1 hour, extracts the keyword field of the exchange information to generate the data definition file of the file to be processed, and continues to use the above embodiment, the data definition file at least comprises the following contents:

table 2 data definition file example

The CHK file is used for file verification before job instantiation, so that the CHK file can be used for platform verification, and the situation that the next processing is triggered when the file is not received is avoided. The generated data definition file can be represented by an xls file, and each item of information in the file is a column value. It should be noted that the processing manner is automatically set according to the actual content of the file to be processed, different values represent different processing manners, and the processing manner in the embodiment of the present disclosure is "1" representing host decoding, which is only an exemplary illustration and does not constitute a limitation of the embodiment of the present disclosure.

Searching a corresponding processing program command and a job type according to the processing mode of each data record, and performing job command definition splicing, wherein the format of the job command definition splicing can be 'processing command + # file source path # + # service date #' (# XXX # is a macro definition and is replaced by valid data in an instantiation stage), so as to generate job definition information, and the corresponding job definition file at least comprises the following contents:

table 3 job definition file example

According to the execution sequence of the job types, the relation between each job name and the corresponding post-job name is recorded in the generated job dependency information, and the job dependency information can be stored in a job dependency definition file with a preset format, and the step represents the completion of the pre-derivation stage of the job.

And the upstream downloads a file to the data exchange platform and then simultaneously downloads a list check file (CHK file), wherein the CHK file records information such as file name, size, service date (date generated by data in the file) and the like for the platform to check. The file verification step may be executed by the data detector, and specifically includes a detection process shown in fig. 4, after the CHK file is verified, the detected actual file information is entered into the source data information file, and the source data information file at least includes the following contents, using the above embodiment:

table 4 source data information file example

In the embodiment of the present disclosure, the actual file information of the file to be processed is recorded in the source data information file, which includes, but is not limited to, the content shown in table 4 above, and the format of the source data information file may be set according to the application requirement, and may be, for example, an xls format file.

Next, polling the source data information file to add a tag to the record to be processed and insert the source data temporary file, searching the corresponding job in the job definition file for the source data temporary file according to the file name and the up and down cursor identifiers, replacing the # file source path # in the job definition file with a file processing directory in the source data information file, replacing the # service date # with a corresponding service log to form an effective job command, setting the instantiated job information to be in a state to be verified, obtaining a job instance of the file to be processed, and correspondingly generating a job instance file, wherein the job instance file at least comprises the following contents in the following manner by using the embodiment:

table 5 job instance file example

Next, the job name in the job instance file is searched for the record with the same name as the post-located job name in the job dependency definition file, dependency instance information is generated, and the following job dependency instance file is correspondingly formed, which at least includes the following contents, along with the above-mentioned embodiment:

table 6 job dependency instance file example

And updating the derivation state of the source data information file record operation added with the label into the derived state, and deleting the temporary file. In the embodiment of the present disclosure, a to-be-processed file is instantiated and then derived, and then a scheduling job of a to-be-processed job is performed, the scheduling of the to-be-processed job may be performed by a scheduling server, one scheduling server is responsible for managing a plurality of job clients, and processing is performed by a plurality of parallel threads, and different parallel threads process the to-be-scheduled and distributed submission processes of the job as shown in fig. 5 and fig. 6, which is not described in detail herein.

It should be noted that, the contents in the files such as the basic information of the download file, the data definition file, the job definition file, the source data information file, and the job instance file shown in the foregoing embodiments are only examples, and in other application scenarios and requirements, the contents may be replaced by other contents, and do not constitute a limitation of the embodiments of the present disclosure.

Fig. 7 schematically shows a block diagram of a data exchange scheduling system according to an embodiment of the present disclosure.

As shown in fig. 7, the data exchange scheduling system 700 includes: a file preprocessing module 710, a job instantiation module 720, and a job scheduling module 730. The system 700 may be used to implement the data exchange scheduling method described with reference to fig. 2.

The file preprocessing module 710 is configured to perform definition preprocessing on a file to be processed, and generate job definition information and job dependency information of the file to be processed. According to an embodiment of the present disclosure, the file preprocessing module 710 may be configured to perform the step S201 described above with reference to fig. 2, for example, and is not described herein again.

And the job instantiation module 720 is configured to perform instantiation processing on the to-be-processed file according to the job definition information to obtain a job instance of the to-be-processed file, and generate dependency instance information of the to-be-processed job according to the job dependency information and the job instance. According to an embodiment of the present disclosure, the job instantiation module 720 may be used to execute the step S202 described above with reference to fig. 2, for example, and will not be described herein again.

And the job scheduling module 730 is configured to determine whether the job to be processed meets the pre-job dependency according to the dependency instance information, and if so, set the job to be processed as a job to be scheduled and perform scheduling processing. According to an embodiment of the present disclosure, the job scheduling module 730 may be configured to perform the step S202 described above with reference to fig. 2, for example, and is not described herein again.

According to an embodiment of the present disclosure, the file preprocessing module 710 is configured to perform definition preprocessing on a file to be processed, and generate job definition information and job dependency information of the file to be processed, including: acquiring basic information of a plurality of files to be processed; carrying out exchange requirement definition on the basic information of the files to be processed to generate exchange information; reading the exchange information within a preset time, and extracting a keyword field of the exchange information to generate a data definition file of the file to be processed; and generating the job definition information according to the processing mode and the job type of each data record in the exchange information, and generating the job dependency information according to the execution sequence of each data record and the job type in the exchange information.

As shown in fig. 8, the system 700 further includes: the document checking module 740 is configured to check the document to be processed, and includes: adding the downloaded directory information in the data definition file into a detection directory table; detecting whether list check files are generated or not by directory information in the detection directory table; if yes, detecting whether the list check file is matched with the data definition file; if the data definition file is matched with the processing directory file, adding the data definition file to the processing directory file, and inputting the detected actual file information into a source data information file; and if not, adding the data definition file to an abnormal processing directory file.

Any number of modules, sub-modules, units, sub-units, or at least part of the functionality of any number thereof according to embodiments of the present disclosure may be implemented in one module. Any one or more of the modules, sub-modules, units, and sub-units according to the embodiments of the present disclosure may be implemented by being split into a plurality of modules. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in any other reasonable manner of hardware or firmware by integrating or packaging a circuit, or in any one of or a suitable combination of software, hardware, and firmware implementations. Alternatively, one or more of the modules, sub-modules, units, sub-units according to embodiments of the disclosure may be at least partially implemented as a computer program module, which when executed may perform the corresponding functions.

For example, any of the file pre-processing module 710, job instantiation module 720, job scheduling module 730, and file verification module 740 may be combined into one module for implementation, or any of them may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of the other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the file preprocessing module 710, the job instantiation module 720, the job scheduling module 730, and the file verification module 740 may be implemented at least partially as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in hardware or firmware in any other reasonable manner of integrating or packaging a circuit, or in any one of three implementations of software, hardware, and firmware, or in any suitable combination of any of them. Alternatively, at least one of the file preprocessing module 710, job instantiation module 720, job scheduling module 730, and file verification module 740 may be implemented at least in part as a computer program module that, when executed, may perform a corresponding function.

The data exchange scheduling method and system provided by the present disclosure may be used in the financial field or other fields, and it should be noted that the data exchange scheduling method and system provided by the present disclosure may be used in the financial field, for example, the scheduling processing after the file of each business system is converted into an operation in the financial field, and may also be used in other fields except the financial field.

Fig. 9 schematically shows a block diagram of an electronic device adapted to implement the above described method according to an embodiment of the present disclosure. The electronic device shown in fig. 9 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.

As shown in fig. 9, the electronic device 900 described in this embodiment includes: a processor 901 which can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)902 or a program loaded from a storage section 908 into a Random Access Memory (RAM) 903. Processor 901 may comprise, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or associated chipset, and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), among others. The processor 901 may also include on-board memory for caching purposes. The processor 901 may comprise a single processing unit or a plurality of processing units for performing the different actions of the method flows according to embodiments of the present disclosure.

In the RAM 903, various programs and data necessary for the operation of the system 900 are stored. The processor 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. The processor 901 performs various operations of the method flows according to the embodiments of the present disclosure by executing programs in the ROM 902 and/or the RAM 903. Note that the programs may also be stored in one or more memories other than the ROM 902 and the RAM 903. The processor 901 may also perform various operations of the method flows according to embodiments of the present disclosure by executing programs stored in the one or more memories.

Electronic device 900 may also include input/output (I/O) interface 905, input/output (I/O) interface 905 also connected to bus 904, according to an embodiment of the present disclosure. The system 900 may also include one or more of the following components connected to the I/O interface 905: an input portion 906 including a keyboard, a mouse, and the like; an output section 907 including components such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 908 including a hard disk and the like; and a communication section 909 including a network interface card such as a LAN card, a modem, or the like. The communication section 909 performs communication processing via a network such as the internet. The drive 910 is also connected to the I/O interface 905 as necessary. A removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 910 as necessary, so that a computer program read out therefrom is mounted into the storage section 908 as necessary.

According to embodiments of the present disclosure, method flows according to embodiments of the present disclosure may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable storage medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 909, and/or installed from the removable medium 911. The computer program, when executed by the processor 901, performs the above-described functions defined in the system of the embodiment of the present disclosure. The systems, devices, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.

An embodiment of the present invention further provides a computer-readable storage medium, which may be included in the apparatus/device/system described in the foregoing embodiment; or may exist separately and not be assembled into the device/apparatus/system. The above-mentioned computer-readable storage medium carries one or more programs which, when executed, implement a data exchange scheduling method according to an embodiment of the present disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In embodiments of the disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, a computer-readable storage medium may include the ROM 902 and/or the RAM 903 described above and/or one or more memories other than the ROM 902 and the RAM 903.

Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for performing the method illustrated in the flow chart. When the computer program product runs in a computer system, the program code is used for causing the computer system to realize the data exchange scheduling method provided by the embodiment of the disclosure.

The computer program performs the above-described functions defined in the system/apparatus of the embodiments of the present disclosure when executed by the processor 901. The systems, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.

In one embodiment, the computer program may be hosted on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted, distributed in the form of a signal on a network medium, and downloaded and installed through the communication section 909 and/or installed from the removable medium 911. The computer program containing program code may be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 909, and/or installed from the removable medium 911. The computer program, when executed by the processor 901, performs the above-described functions defined in the system of the embodiment of the present disclosure. The systems, devices, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.

In accordance with embodiments of the present disclosure, program code for executing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, these computer programs may be implemented using high level procedural and/or object oriented programming languages, and/or assembly/machine languages. The programming language includes, but is not limited to, programming languages such as Java, C + +, python, the "C" language, or the like. The program code may execute entirely on the user computing device, partly on the user device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).

It should be noted that each functional module in each embodiment of the present disclosure may be integrated into one processing module, or each module may exist alone physically, or two or more modules are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be substantially or partially embodied in the form of a software product, or all or part of the technical solution that contributes to the prior art.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Those skilled in the art will appreciate that various combinations and/or combinations of features recited in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not expressly recited in the present disclosure. In particular, various combinations and/or combinations of the features recited in the various embodiments and/or claims of the present disclosure may be made without departing from the spirit or teaching of the present disclosure. All such combinations and/or associations are within the scope of the present disclosure.

While the disclosure has been shown and described with reference to certain exemplary embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the disclosure as defined by the appended claims and their equivalents. Accordingly, the scope of the present disclosure should not be limited to the above-described embodiments, but should be defined not only by the appended claims, but also by equivalents thereof.

Claims

1. A method for scheduling data exchange, comprising:

performing definition preprocessing on a file to be processed to generate job definition information and job dependency information of the file to be processed;

according to the job definition information, the to-be-processed file is instantiated to obtain a job instance of the to-be-processed file, and according to the job dependency information and the job instance, dependency instance information of the to-be-processed job is generated;

and judging whether the job to be processed meets the dependency of the front job or not according to the dependency example information, and if so, setting the job to be processed as the job to be scheduled and carrying out scheduling processing.

2. The data exchange scheduling method according to claim 1, wherein the performing the definition preprocessing on the file to be processed to generate the job definition information and the job dependency information of the file to be processed includes:

acquiring basic information of a plurality of files to be processed;

carrying out exchange requirement definition on the basic information of the files to be processed to generate exchange information;

reading the exchange information within a preset time, and extracting a keyword field of the exchange information to generate a data definition file of the file to be processed;

and generating the job definition information according to the processing mode and the job type of each data record in the exchange information, and generating the job dependency information according to the execution sequence of each data record and the job type in the exchange information.

3. The data exchange scheduling method according to claim 2, wherein before the instantiation process is performed on the file to be processed according to the job definition information, the method further comprises:

adding the downloaded directory information in the data definition file into a detection directory table;

detecting whether a list verification file is generated or not by detecting the directory information in the detection directory table;

if yes, detecting whether the list check file is matched with the data definition file;

if the data definition file is matched with the processing directory file, adding the data definition file to the processing directory file, and inputting the detected actual file information into a source data information file; and if not, adding the data definition file to an abnormal processing directory file.

4. The data exchange scheduling method according to claim 3, wherein the instantiating the file to be processed according to the job definition information to obtain the job instance of the file to be processed includes:

and matching the file to be processed recorded in the source data information file with the job definition information and the data definition file, and converting the replacement macro into an example job to obtain a job example of the file to be processed.

5. The data exchange scheduling method according to claim 4, wherein the generating dependent instance information of the job to be processed according to the job dependent information and the job instance comprises:

and generating the dependency instance information of the job to be processed according to the job name in the job instance file and the post job name in the job dependency information.

6. The data exchange scheduling method according to claim 1, wherein the determining whether the job to be processed satisfies a pre-job dependency comprises:

judging whether the job to be processed has a pre-job dependency or not;

if not, setting the job to be processed as the job to be scheduled;

if yes, judging whether the front operation is processed or not, and if the front operation is processed, setting the operation to be processed as the operation to be scheduled.

7. The data exchange scheduling method of claim 6, further comprising:

judging whether the job to be scheduled is a dead job, wherein the dead job represents the job to be scheduled which is not subjected to job processing within a preset time;

if so, resetting the dead operation as the operation to be scheduled.

8. The data exchange scheduling method according to claim 6, wherein the setting the job to be processed as a job to be scheduled and performing scheduling processing includes:

adding the job to be scheduled into a distribution job queue;

sequentially taking out a plurality of jobs to be scheduled from the distribution job queue and distributing the jobs to be scheduled to idle node addresses in the job client, and setting the state of the jobs to be scheduled to be processed;

and submitting the plurality of jobs to be scheduled to the job client for job processing, and updating the job state of the jobs to be processed after processing.

9. A data exchange scheduling system, comprising:

the file preprocessing module is used for performing definition preprocessing on a file to be processed and generating job definition information and job dependency information of the file to be processed;

the job instantiation module is used for instantiating the file to be processed according to the job definition information to obtain a job instance of the file to be processed and generating dependency instance information of the job to be processed according to the job dependency information and the job instance;

and the job scheduling module is used for judging whether the job to be processed meets the requirement of the front job according to the dependency example information, and if so, setting the job to be processed as the job to be scheduled and performing scheduling processing.

10. The data exchange scheduling system of claim 9, wherein the file preprocessing module is configured to perform definition preprocessing on a file to be processed, and generate job definition information and job dependency information of the file to be processed, and the method includes:

acquiring basic information of a plurality of files to be processed;

11. The data exchange scheduling system of claim 9, further comprising:

the file inspection module is used for inspecting the file to be processed and comprises: adding the downloaded directory information in the data definition file into a detection directory table; detecting whether list check files are generated or not by directory information in the detection directory table; if yes, detecting whether the list check file is matched with the data definition file; if the data definition file is matched with the processing directory file, adding the data definition file to the processing directory file, and inputting the detected actual file information into a source data information file; and if not, adding the data definition file to an abnormal processing directory file.

12. An electronic device, comprising: memory, processor and computer program stored on the memory and executable on the processor, characterized in that the processor implements the data exchange scheduling method according to any of claims 1 to 8 when executing the computer program.

13. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out a data exchange scheduling method according to any one of claims 1 to 8.

14. A computer program product comprising a computer program which, when executed by a processor, implements a data exchange scheduling method as claimed in any one of claims 1 to 8.