CN114500318A

CN114500318A - Batch operation monitoring method and device, equipment and medium

Info

Publication number: CN114500318A
Application number: CN202210127498.2A
Authority: CN
Inventors: 曹琎; 王亚军; 陈万红; 邱枫
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2022-02-11
Filing date: 2022-02-11
Publication date: 2022-05-13
Anticipated expiration: 2042-02-11
Also published as: CN114500318B

Abstract

The utility model provides a batch job monitoring method, which is applied to the technical field of finance and computers, and comprises the following steps: receiving a job message from an upstream node in a batch job link; updating the job message of the current node in the batch job link by using the job message from the upstream node, wherein the current node is positioned at the downstream of the upstream node in the batch job link; acquiring the operation duration of the current node according to the updated operation message of the current node; and sending out early warning information under the condition that the operation duration of the current node meets the preset time threshold condition. The present disclosure also provides a batch job monitoring apparatus, a device, a storage medium, and a program product.

Description

Batch operation monitoring method and device, equipment and medium

Technical Field

The present disclosure relates to the field of financial and computer technologies, and in particular, to a batch job monitoring method, apparatus, device, medium, and program product.

Background

With the continuous improvement of the informatization degree of financial industries such as banks and the like and the continuous increase of the number of users, the data volume generated by the background server is greatly increased. In addition, along with the continuous upgrading and adjustment of background architectures of information technology departments such as banks and the like, the complexity of integration and information interaction among application systems is continuously improved, and a large number of application systems complete data interaction synchronization through batch operation, so that higher requirements are provided for the performance processing requirements of the batch operation. In the transmission of batch jobs, due to reasons such as network interruption, size of transmitted files, performance processing capability of batch node servers, format of batch files and the like, the delay and even interruption of batch jobs can be caused, and batch file transmission and information interaction among application systems are influenced. In the current technical scheme, the operation node cannot sense the operation state of the upstream and downstream nodes in the batch operation link, the batch operation efficiency is influenced, and the operation and maintenance cost is increased.

Disclosure of Invention

In view of the foregoing, the present disclosure provides a batch job monitoring method, apparatus, device, medium, and program product based on a packet.

In a first aspect of the present disclosure, a batch job monitoring method is provided, including:

receiving a job message from an upstream node in a batch job link;

updating the job message of the current node in the batch job link by using the job message from the upstream node, wherein the current node is positioned at the downstream of the upstream node in the batch job link;

acquiring the operation duration of the current node according to the updated operation message of the current node;

and sending out early warning information under the condition that the operation duration of the current node meets the preset time threshold condition.

According to the embodiment of the disclosure, the operation message comprises a batch operation project name, an operation node name, an operation file size, accumulated operation time and an error reporting node name;

the size of the job file represents the size of a file to be processed by a job node;

the accumulated operation duration represents the accumulated operation time of the batch operation link;

and the error-reported node name is used for recording the node name of the operation interruption in the batch operation link.

According to an embodiment of the present disclosure, the updating the job packet of the current node in the batch job link by using the job packet from the upstream node includes at least one of: and updating the size of the job file of the current node, the accumulated job duration of the current node and the error-reporting node name of the current node.

According to an embodiment of the present disclosure, the updating the size of the job file of the current node includes:

acquiring the size of the job file of each upstream node in the batch job link through the current node, and accumulating to obtain an accumulated value of the size of the job file;

acquiring the size of a forwarding job file through a historical log file of a current node, wherein the historical log file is used for recording historical job information of the current node, and comprises the total number of the historical job files, the processing duration of the historical jobs and the size of the forwarding job file sent to a downstream node;

and processing the accumulated value of the sizes of the job files and the size of the forwarded job file by using a local weighted linear regression algorithm, and updating the size of the job file of the current node.

According to an embodiment of the present disclosure, the updating the accumulated operation duration of the current node includes:

acquiring historical job processing duration through a historical log file of a current node;

processing the historical job processing duration and the job file size of the current node through a local weighted linear regression algorithm, and determining the processing duration of the current job node;

acquiring the accumulated operation duration of each upstream node in the batch operation link through the current node and acquiring the maximum value of the accumulated operation duration of the upstream node;

and calculating according to the processing time length of the current operation node and the maximum value of the accumulated operation time length of the upstream node, and updating the accumulated operation time length of the current node.

According to an embodiment of the present disclosure, the updating the error-reporting node name of the current node includes:

detecting the operation state of the upstream node in the batch operation link in real time by using the current node;

under the condition that the upstream node has the job interruption, updating the error-reporting node name of the current node to the name of the upstream node having the job interruption;

transmitting the error-reported node name to a downstream node in a batch operation link by using an operation message through a current node;

and under the condition that the node with the job interruption in the batch job link is recovered, weighing the error-reported node name in the job message of the current job node to be null.

According to an embodiment of the present disclosure, the sending out the warning information when the operation duration of the current node meets the condition of the preset time threshold includes:

under the condition that the maximum value of the accumulated operation time length of the upstream nodes in the batch operation link exceeds a preset operation starting time threshold value of the current node, first early warning information is sent out through the current node;

and under the condition that the job processing duration of the current node exceeds a job processing time threshold preset by the current node, sending second early warning information through the current node.

According to an embodiment of the present disclosure, the batch job monitoring method further includes:

and sending the updated job message of the current node to a downstream node in the batch job link.

In a second aspect of the present disclosure, there is provided a batch job monitoring apparatus, including:

the receiving module is used for receiving the operation message from the upstream node in the batch operation link;

the updating module is used for updating the job message of the current node in the batch job link by using the job message from the upstream node, wherein the current node is positioned at the downstream of the upstream node in the batch job link;

the acquisition module is used for acquiring the operation duration of the current node according to the updated operation message of the current node;

and the early warning module is used for sending out early warning information under the condition that the operation time of the current node meets the preset time threshold condition.

In a third aspect of the present disclosure, there is provided an electronic device including:

one or more processors;

a storage device for storing one or more programs,

wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the batch job monitoring method described above.

In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, having executable instructions stored thereon, which when executed by a processor, cause the processor to perform the batch job monitoring method described above.

In a fifth aspect of the disclosure, a computer program product is provided, comprising a computer program, which when executed by a processor, implements the batch job monitoring method described above.

According to the batch job monitoring method, the batch job monitoring device, the equipment, the medium and the computer program product based on the message, the job state information of the upstream and downstream nodes is transmitted in the batch job link through the message, so that the efficiency of batch job is improved; meanwhile, related fields in the message are optimized, early warning can be performed in advance, and operation and maintenance personnel can conveniently perform batch operation management.

Drawings

The foregoing and other objects, features and advantages of the disclosure will be apparent from the following description of embodiments of the disclosure, which proceeds with reference to the accompanying drawings, in which:

FIG. 1 schematically illustrates an application scenario diagram of a batch job monitoring method according to an embodiment of the present disclosure;

FIG. 2 schematically illustrates a flow diagram of a batch job monitoring method according to an embodiment of the present disclosure;

FIG. 3 schematically illustrates a flow chart for updating a job file size of a current node according to an embodiment of the present disclosure;

FIG. 4 schematically illustrates a flow chart for updating an accumulated job time for a current node, implemented in accordance with the present disclosure;

FIG. 5 is a schematic diagram that schematically illustrates current node historical job durations, in accordance with an embodiment of the present disclosure;

FIG. 6 schematically shows a weighting factor k according to an embodiment of the disclosure₂A fitted graph of 1;

FIG. 7 schematically shows a weighting factor k according to an embodiment of the disclosure₂Fit plot of 0.01;

FIG. 8 schematically shows a weighting factor k according to an embodiment of the disclosure₂Fit plot of 0.003;

FIG. 9 schematically illustrates a flow diagram for updating an error-flagged node name for a current node according to an embodiment of the present disclosure;

FIG. 10 is a schematic diagram illustrating a process for error reporting node delivery in a bulk job link according to an embodiment of the present disclosure;

FIG. 11 schematically illustrates a flow diagram of a batch job based on job messages, according to an embodiment of the disclosure;

fig. 12 is a block diagram schematically illustrating a structure of a batch job monitoring apparatus according to an embodiment of the present disclosure;

FIG. 13 schematically illustrates a block diagram of an electronic device suitable for implementing a batch job monitoring method according to an embodiment of the present disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.

Where a convention analogous to "at least one of A, B and C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B and C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).

The main characteristics of batch operation are: the method has the advantages of large batch file quantity, large batch file generally, low timeliness, serial processing related to multiple system applications and the like. The processing time of the batch job of one upstream node in the batch link is too long, or the processing time of all the batch link nodes at the downstream is affected by the processing time or the interruption. In the current technical scheme, an upstream batch file arrival time threshold and a batch processing aging threshold of the node are configured on the node mainly according to production experience values. The problems that exist are that: the nodes in the batch link can only sense the current actual processing condition of the batch files of the node, and cannot sense batch abnormity occurring at the upstream. When the arrival time of the upstream batch files exceeds a threshold value or the processing timeliness of the node exceeds the threshold value, the node can give an alarm for batch operation and inform operation and maintenance personnel to process the files, and therefore higher time processing cost is brought to the node, particularly the downstream node for processing the batch operation.

In view of the above, the present disclosure provides a batch job monitoring method, a batch job monitoring apparatus, a device, a storage medium, and a program product. According to the batch job monitoring method, the job state information of the upstream node in the batch job link and the historical log file of the current node are effectively utilized, the job duration and the job file size of the current node can be effectively estimated, and therefore the purposes of early warning and operation and maintenance cost saving are achieved.

In the technical scheme of the disclosure, the acquisition, storage, application and the like of the personal information of the related user all accord with the regulations of related laws and regulations, necessary security measures are taken, and the customs of the public order is not violated.

The embodiment of the disclosure provides a batch job monitoring method which can be applied to the fields including but not limited to finance.

Fig. 1 schematically shows an application scenario diagram of a batch job monitoring method according to an embodiment of the present disclosure.

As shown in fig. 1, an application scenario 100 according to this embodiment may include

terminal devices

101, 102, 103, a network 104, and

servers

105, 106, 107. The network 104 serves as a medium for providing communication links between the

terminal devices

101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

The user may use the

terminal devices

101, 102, 103 to interact with the

servers

105, 106, 107 via the network 104 to receive or send messages or the like. The

terminal devices

101, 102, 103 may have installed thereon various communication client applications, such as shopping-like applications, web browser applications, search-like applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).

The

terminal devices

101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The

servers

105, 106, 107 may be servers providing various services, such as a background management server (for example only) providing support for websites browsed by users using the

terminal devices

101, 102, 103. The background management server may analyze and perform other processing on the received data such as the user request, and feed back a processing result (e.g., a webpage, information, or data obtained or generated according to the user request) to the terminal device.

It should be noted that the batch job monitoring method based on message provided by the embodiment of the present disclosure may be generally executed by the

servers

105, 106, and 107. Accordingly, the message-based batch job monitoring system provided by the embodiments of the present disclosure may be generally disposed in the

servers

105, 106, 107. The batch job monitoring method based on the message provided by the embodiment of the present disclosure may also be executed by a server or a server cluster different from the

servers

105, 106, and 107 and capable of communicating with the

terminal devices

101, 102, and 103 and/or the server 105. Correspondingly, the batch job monitoring system based on the message provided by the embodiment of the present disclosure may also be disposed in a server or a server cluster which is different from the

servers

105, 106, 107 and can communicate with the

terminal devices

101, 102, 103 and/or the

servers

105, 106, 107.

It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

The following describes a message-based batch job monitoring method according to an embodiment of the present disclosure in detail with reference to fig. 2 to 11 based on the scenario described in fig. 1.

FIG. 2 schematically shows a flow diagram of a batch job monitoring method according to an embodiment of the disclosure.

As shown in fig. 2, the method for monitoring batch jobs based on messages according to this embodiment includes operations S210 to S240.

In operation S210, a job packet from an upstream node in a bulk job link is received.

The operation message is used for recording the operation state of each operation node in the batch operation link, and the operation message can effectively transmit the operation information in the batch operation link, so that the efficiency of batch operation can be improved.

In operation S220, a job packet of a current node in the batch job link is updated with the job packet from the upstream node, wherein the current node is located downstream of the upstream node in the batch job link.

The current node in the batch operation link can update the operation message of the current node in time according to the operation message of the upstream node.

In operation S230, the operation duration of the current node is obtained according to the updated operation packet of the current node.

The job duration may include, for example, a processing duration for the current node to process the job file and a maximum value of the accumulated job durations of the nodes upstream.

In operation S240, in a case that the operation duration of the current node meets a preset time threshold condition, early warning information is sent out.

And comparing the current node with a preset time threshold, and sending early warning information to related operation and maintenance personnel when the operation time of the current node exceeds the preset time threshold.

According to the batch job monitoring method disclosed by the embodiment, the job state information of the job nodes is transmitted on the batch job chain through the message, so that the upstream and downstream nodes in the batch job chain can be effectively monitored, and the time processing cost is greatly reduced.

and the error-reporting node name is used for recording the node name of the operation interruption in the batch operation link.

The job message is further described in detail with reference to table 1.

As shown in table 1, table 1 schematically shows the format of the job packet in the batch job link and the meaning of each field in the job packet.

TABLE 1 Job message

The batchName represents a batch operation project name, namely a batch project name needing to be added with monitoring in a link; the batchNode represents the name of the operation Node, namely the name of the Node sending the message to the downstream, such as Node [ i ], Node [ j ]; the batchPatkSize represents the size of a job file of a job node, namely the total size of batch files needing to be transmitted to the downstream, wherein the current node pre-estimates the total size of the batch files needing to be transmitted to the downstream by the current node in real time; the accumulatime represents the accumulated duration of the operation in the batch operation link ending to the current node, and the field takes the maximum value of the upstream field under the condition that the current node has a plurality of upstream nodes; the error BatchNode represents the name of the error-reporting node, namely, the name of the node is transmitted to the downstream after a certain node at the upstream is interrupted in batches, and the name of the node is transmitted to the downstream node in sequence; this field is updated to null after the upstream node interrupt resumes.

The value of the batchPacketSize field read to the upstream batchPacketSize field at the current node changes, and the value of the batchPacketSize sent to the downstream by the current node changes with the historical experience value. The value of batchPacketSize may not be less than the total current actual file size of the current node.

The accumulateTime field receives heartbeat messages (i.e., the operation messages of the present disclosure) of each upstream node at a current node, reads and accumulates the batchPacketSize field of each node message, estimates an aging processTime (i.e., the processing time of the present disclosure) of a batch file required to be processed by the current node according to a historical experience value, adds the aging processTime to the value of the received upstream accumulateTime (generally, the maximum value of each upstream node), writes the value into the accumulateTime field sent to the downstream by the current node, wherein the value of the upstream batchPacketSize field read by the current node changes, the estimated processing time of the batch file by the current node changes along with the historical experience value, the value of the downstream accumulateTime changes, and the value of the accumulateTime cannot be smaller than the current actual time of the current node; and if the batch operation of the current node is interrupted and the redo is taken over, the redo time and the estimated batch processing time processTime of the current node are used as the accumlatime and sent to the downstream node.

The information in the batch operation link is transmitted through the operation message provided by the disclosure, and the operation information and the state of the operation node can be timely and effectively transmitted through the updating of the operation message, so that the operation and maintenance efficiency is improved.

FIG. 3 schematically shows a flowchart for updating the job file size of the current node according to an embodiment of the present disclosure.

As shown in fig. 3, updating the job file size of the current node includes operations S310 to S330.

In operation S310, the job file size of each upstream node in the batch job link is obtained through the current node and accumulated, so as to obtain a job file size accumulated value.

The current node reads the batchPatchSize field of the upstream node through the received job message (or heartbeat message), accumulates the batchPatchSize field, estimates the size of the batch files needing to be transmitted downstream by the current node, and writes the batchPatchSize field of the job message sent to the downstream node.

In operation 320, the size of the forwarding job file is obtained through a history log file of the current node, where the history log file is used to record history job information of the current node, and the history log file includes a total number of history job files, a processing time length of the history job, and a size of the forwarding job file sent to a downstream node.

When the batchPacketSize field of the job message sent to the downstream node is estimated, the current node can fully utilize the job history log file of the current node to estimate the size of the job file forwarded to the downstream node.

In operation S330, the job file size of the current node is updated by processing the accumulated value of the job file size and the forwarded job file size using a locally weighted linear regression algorithm.

Wherein the local weighted linear regression algorithm is represented by formula (1) and formula (2):

wherein, X₁Is a feature matrix comprising samples x₁In matrix form, y₁Is a target variable vector, W₁The weight corresponding to the Gaussian kernel is represented by a weight matrix of diagonal elements and a weight coefficient k₁And selecting a relative optimal value according to the learning result for the user specification.

According to the method for updating the size of the job file, provided by the embodiment of the disclosure, the historical log file of the current node can be fully utilized, so that the size of the job file can be estimated more accurately.

Table 2 is a table of historical log files according to an embodiment of the present disclosure, and the historical log files are further described below with reference to table 2.

Table 2 History Log File Table

As shown in table 2, the current node maintains a history log file table; wherein, totalSize represents the total number of the historical operation files of the current Node, reads the heartbeat message of the executed upstream batch operation as the effective input, and records the execution duration of the batch operation of the current Node and the size of the batch file of each Node needing to be transmitted to the downstream as the historical record value by taking the total size of the batch import file of each Node such as the upstream Node [ i ], Node [ j ], Node [ k ] and the like as parameters. Periodically using the data in the history table for machine learning, and using an algorithm such as Local Weighted Linear Regression (LWLR) to fit the values of the relevant fields in the history log file table. When historical experience does not cover values, a default a priori value may be used.

FIG. 4 schematically illustrates a flow chart for updating a cumulative job duration for a current node, implemented in accordance with the present disclosure.

As shown in fig. 4, the method for updating the accumulated job duration in the job message by the current node includes operations S410 to S440.

In operation S410, a history job processing time length is acquired through a history log file of the current node.

In operation S420, the historical job processing duration and the job file size of the current node are processed through a local weighted linear regression algorithm, and the processing duration of the current job node is determined.

Wherein the locally weighted linear regression algorithm is determined by formula (3) and formula (4),

wherein, X₂Is a feature matrix, y₂Is the target variable vector, W₂Is a Gaussian kernel, is a weight matrix of diagonal elements, and has a weight coefficient k₂And selecting a relative optimal value according to the learning result for the user specification.

In operation S430, the accumulated job time length of each upstream node in the batch job link is obtained through the current node and the maximum value of the accumulated job time length of the upstream node is obtained.

When there are a plurality of upstream nodes, the current node selects the upstream node with the largest accumulated job time length as the initial value for updating the accumulated job time length of the current node.

In operation S440, the operation is performed by using the maximum value of the processing duration of the current job node and the accumulated job duration of the upstream node, and the accumulated job duration of the current node is updated.

The flow of updating the accumulated operation duration field in the operation message by the current node will be further described in detail with reference to fig. 5 to 8.

FIG. 5 is a schematic diagram that schematically illustrates current node historical job durations, in accordance with an embodiment of the present disclosure.

As shown in fig. 5, the X axis represents the total number of imported history job files, the Y axis represents the history job time length, and it can be seen from fig. 5 that there is a certain linear relationship between the history job time length and the total number of history job files, and the relationship between the history job time length and the total number of history job files can be fitted by using a local weighted linear regression algorithm.

FIG. 6 schematically shows a weighting factor k according to an embodiment of the disclosure₂A fit of 1.

As shown in FIG. 6, the X-axis represents the total number of job files imported, the Y-axis represents the job duration, and it can be seen from FIG. 6 that when k is₂When 1, the sample input cannot be accurately fitted.

FIG. 7 schematically shows a weighting factor k according to an embodiment of the disclosure₂Fit plot of 0.01.

As shown in FIG. 7, the X-axis represents the total number of job files imported, the Y-axis represents the duration of the job, and it can be seen from FIG. 7 that when k is₂When the value is 0.01, the sample input can be accurately fitted.

FIG. 8 schematically shows a weighting factor k according to an embodiment of the disclosure₂Fit of 0.003.

As shown in FIG. 8, the X-axis represents the total number of job files imported, the Y-axis represents the duration of the job, and it can be seen from FIG. 8 that when k is₂At 0.003, there is an overfitting phenomenon, and the fitted curve cannot adapt to the change of the actual input variable in use.

Comparing the fitting chart 6-8 with the actual operation duration chart 5 when k is₂0.01 enables an accurate fit of the relationship between job file size and job duration.

FIG. 9 schematically illustrates a flow diagram for updating an error-reported node name for a current node according to an embodiment of the disclosure.

As shown in fig. 9, the method for updating the error-reported node name by the current node includes operations S910 to S940.

In operation S910, a job status of an upstream node in the batch job link is detected in real time using the current node.

In operation S920, in the case where a job interrupt occurs at an upstream node, the error-reported node name of the current node is updated to the name of the upstream node at which the job interrupt occurs.

In operation S930, the error-reporting node name is propagated to the downstream nodes in the batch job link by the current node using the job packet.

In operation S940, in the case that the node in which the job interrupt occurs in the batch job link is recovered, the error-reporting node name weight in the job message of the current job node is set to null.

Through the update of the wrong node name reported in the operation message of the current node, the node with the operation fault in the batch operation link can be timely sensed by the downstream node, and therefore the early warning efficiency can be improved.

FIG. 10 is a diagram schematically illustrating a process for an error reporting node passing in a bulk job link, according to an embodiment of the present disclosure.

As shown in fig. 10, in the batch job link, the Node [ i ] in the Node is sent to the downstream Node [ i +1] as a heartbeat message (i.e., the job message of the present disclosure) configured upstream, and the downstream Node [ i +1] monitors the message. If the upstream Node [ i ] has batch interruption, recording the heartbeat message sent to the Node [ i +1] by the Node [ i ], and indicating that the Node [ i ] has batch interruption. Node [ i +1] sends heartbeat message to downstream Node [ i +2], indicating that Node [ i ] has batch interruption. Node [ i +2] receives the information and informs Node [ i ] of batch interruption downstream through the heartbeat message. And similarly, after the batch interruption is recovered, the interruption content corresponding to the heartbeat message sent by the upstream node is emptied, and the downstream node synchronously updates the interruption alarm of the node according to the content in the upstream message.

According to the embodiment of the disclosure, when the operation duration of the current node meets the condition of the preset time threshold, the sending of the early warning information includes: under the condition that the maximum value of the accumulated operation time length of the upstream nodes in the batch operation link exceeds a preset operation starting time threshold value of the current node, first early warning information is sent out through the current node; and under the condition that the job processing duration of the current node exceeds a job processing time threshold preset by the current node, sending second early warning information through the current node.

Taking Node [ i +1] as the current Node, the process of sending the early warning information is further explained. The current Node is provided with a threshold of batch start time after the files arrive at the same time, and if the estimated link accumulated processing time (accumulatimettime) transmitted by a certain upstream Node [ j ] heartbeat message (namely, the operation message of the disclosure) of the current Node exceeds the threshold of the start time of the current Node, the current Node gives an early warning that the batch start time is too late, so that an administrator of an application system of the current Node can evaluate the influence.

If the estimated batch file size batchPacketSize transmitted by the heartbeat message of one upstream Node [ k ] of the current Node is superposed with the batch PacketSize transmitted by the Node [ i ] and the Node [ j ], the total size (namely the size of the job file of the disclosure) of the batch job files of the current Node is obtained, and the total size exceeds the execution duration threshold of the current Node through historical experience evaluation, the early warning batch execution time of the current Node is too long, so that the influence of the administrator of the application system of the current Node is evaluated.

According to an embodiment of the present disclosure, the batch job monitoring method further includes: and sending the updated job message of the current node to a downstream node in the batch job link.

According to the batch job monitoring method disclosed by the embodiment of the disclosure, the job status of the upstream and downstream nodes in the batch job link is efficiently transmitted through the job message, so that the real-time sharing of the job status of the upstream and downstream nodes is realized, the efficiency of batch job can be improved, and the operation and maintenance time and the operation and maintenance cost can be saved.

Fig. 11 schematically shows a flowchart of a batch job based on a job packet according to an embodiment of the present disclosure, and the above batch job monitoring method is further described in detail with reference to fig. 11.

As shown in fig. 11, the current Node [ i +1] receives job messages of a plurality of upstream nodes (e.g., Node [ i ], Node [ j ], and Node [ k ]), reads the size of the pre-estimated batch processing file sent to the current Node by each upstream Node, and reads the batch processing time end time of each upstream Node, i.e., the accumulated job duration in the job message. When the operation time of each upstream node exceeds the time threshold value of the current node for starting operation, early warning information is sent to operation and maintenance personnel, namely, the operation time is too late, and the operation and maintenance personnel are requested to perform operation scheduling in time.

The current node reads the size of the job file of each upstream node, and the LWLR is utilized to estimate the size of the job file of the current node and estimate the job processing time length.

And when the operation processing time estimated by the current node exceeds a preset time threshold, sending early warning information to operation and maintenance personnel so as to facilitate the operation and maintenance personnel to process.

And sending the updated job message of the current node to each downstream node.

Fig. 12 schematically shows a block diagram of a batch job monitoring apparatus according to an embodiment of the present disclosure.

As shown in fig. 12, the batch job monitoring apparatus 1200 of this embodiment includes a receiving module 1210, an updating module 1220, an obtaining module 1230, and an early warning module 1240.

A receiving module 1210, configured to receive a job packet of an upstream node in a batch job link;

an updating module 1220, configured to update the job packet of the current node in the batch job link by using the job packet of the upstream node;

an obtaining module 1230, configured to obtain the job time of the current node according to the updated job packet of the current node;

and the early warning module 1240 is used for sending out early warning information when the working time of the current node exceeds a preset time threshold.

The monitoring device for batch operation can configure health check from upstream to downstream for the link nodes of batch operation, and realize early warning of batch operation risks through message transmission interaction among the batch nodes. The device can early warn the administrator of the downstream application system of the transmission condition of the batch files in advance. And abundant time is provided for downstream to process batch operation ahead of time. Therefore, it is expected that the overall operating cost can be effectively reduced.

According to the embodiment of the present disclosure, any plurality of the receiving module 1210, the updating module 1220, the obtaining module 1230 and the early warning module 1240 may be combined into one module to be implemented, or any one of the modules may be split into a plurality of modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of the other modules and implemented in one module. At least one of the receiving module 1210, the updating module 1220, the obtaining module 1230 and the early warning module 1240 according to the embodiments of the present disclosure may be implemented at least partially as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in hardware or firmware in any other reasonable manner of integrating or packaging a circuit, or may be implemented in any one of three implementations of software, hardware and firmware, or in a suitable combination of any of them. Alternatively, at least one of the receiving module 1210, the updating module 1220, the obtaining module 1230 and the early warning module 1240 may be at least partially implemented as a computer program module, which when executed may perform the corresponding functions.

As shown in fig. 13, an electronic device 1300 according to an embodiment of the present disclosure includes a processor 1301 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)1302 or a program loaded from a storage section 1308 into a Random Access Memory (RAM) 1303. The processor 1301 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or associated chipset, and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), among others. The processor 1301 may also include onboard memory for caching purposes. Processor 1301 may include a single processing unit or multiple processing units for performing the different actions of the method flows according to embodiments of the present disclosure.

In the RAM 1303, various programs and data necessary for the operation of the electronic apparatus 1300 are stored. The processor 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. The processor 1301 performs various operations of the method flows according to the embodiments of the present disclosure by executing programs in the ROM 1302 and/or the RAM 1303. Note that the programs may also be stored in one or more memories other than the ROM 1302 and RAM 1303. The processor 1301 may also perform various operations of the method flows according to embodiments of the present disclosure by executing programs stored in the one or more memories.

Electronic device 1300 may also include input/output (I/O) interface 1305, which is also connected to bus 1304, according to an embodiment of the present disclosure. The electronic device 1300 may also include one or more of the following components connected to the I/O interface 1305: an input portion 1306 including a keyboard, a mouse, and the like; an output section 1307 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 1308 including a hard disk and the like; and a communication section 1309 including a network interface card such as a LAN card, a modem, or the like. The communication section 1309 performs communication processing via a network such as the internet. A drive 1310 is also connected to the I/O interface 1305 as needed. A removable medium 1311 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 1310 as necessary, so that a computer program read out therefrom is mounted into the storage portion 1308 as necessary.

The present disclosure also provides a computer-readable storage medium, which may be contained in the apparatus/device/system described in the above embodiments; or may exist separately and not be assembled into the device/apparatus/system. The computer-readable storage medium carries one or more programs which, when executed, implement the method according to an embodiment of the disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, a computer-readable storage medium may include one or more memories other than the ROM 1302 and/or the RAM 1303 and/or the ROM 1302 and the RAM 1303 described above.

Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for performing the method illustrated in the flow chart. When the computer program product runs in a computer system, the program code is used for causing the computer system to realize the item recommendation method provided by the embodiment of the disclosure.

The computer program performs the above-described functions defined in the system/apparatus of the embodiments of the present disclosure when executed by the processor 1301. The systems, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.

In one embodiment, the computer program may be hosted on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted in the form of a signal on a network medium, distributed, downloaded and installed via communications component 1309, and/or installed from removable media 1311. The computer program containing program code may be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

In such embodiments, the computer program may be downloaded and installed from a network via communications component 1309 and/or installed from removable media 1311. The computer program, when executed by the processor 1301, performs the functions defined in the system of the embodiments of the present disclosure. The systems, devices, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.

In accordance with embodiments of the present disclosure, program code for executing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, these computer programs may be implemented using high level procedural and/or object oriented programming languages, and/or assembly/machine languages. The programming language includes, but is not limited to, programming languages such as Java, C + +, python, the "C" language, or the like. The program code may execute entirely on the user computing device, partly on the user device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Those skilled in the art will appreciate that various combinations and/or combinations of features recited in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not expressly recited in the present disclosure. In particular, various combinations and/or combinations of the features recited in the various embodiments and/or claims of the present disclosure may be made without departing from the spirit or teaching of the present disclosure. All such combinations and/or associations are within the scope of the present disclosure.

The embodiments of the present disclosure have been described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described separately above, this does not mean that the measures in the embodiments cannot be used in advantageous combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be devised by those skilled in the art without departing from the scope of the present disclosure, and such alternatives and modifications are intended to be within the scope of the present disclosure.

Claims

1. A batch job monitoring method, comprising:

receiving a job message from an upstream node in a batch job link;

and sending out early warning information under the condition that the operation duration of the current node meets the condition of a preset time threshold.

2. The method of claim 1, wherein the job message includes a batch job project name, a job node name, a job file size, an accumulated job duration, and an error node name;

wherein the accumulated operation duration represents the accumulated operation time of the batch operation link;

and the error reporting node name is used for recording the node name of the operation interruption in the batch operation link.

3. The method of claim 2, wherein said updating the job message of the current node in the bulk job link with the job message from the upstream node comprises at least one of: and updating the size of the job file of the current node, the accumulated job duration of the current node and the error-reporting node name of the current node.

4. The method of claim 3, wherein the updating the job file size of the current node comprises:

acquiring the size of a forwarding job file through a historical log file of the current node, wherein the historical log file is used for recording historical job information of the current node, and the historical log file comprises the total number of the historical job files, the processing duration of the historical jobs and the size of the forwarding job file sent to a downstream node;

and processing the accumulated value of the sizes of the operation files and the size of the forwarded operation file by using a local weighted linear regression algorithm, and updating the size of the operation file of the current node.

5. The method of claim 3, wherein said updating the cumulative job duration for the current node comprises:

acquiring historical job processing duration through the historical log file of the current node;

6. The method of claim 3, wherein the updating the error-reporting node name of the current node comprises:

under the condition that the upstream node has operation interruption, updating the error-reported node name of the current node to the name of the upstream node having operation interruption;

transmitting the error-reported node name to a downstream node in a batch operation link by using an operation message through the current node;

and under the condition that the node with the operation interruption in the batch operation link is recovered, weighing the name of the error-reporting node in the operation message of the current operation node to be null.

7. The method according to claim 2, wherein when the working duration of the current node meets a preset time threshold condition, sending out early warning information:

under the condition that the maximum value of the accumulated operation time length of the upstream nodes in the batch operation link exceeds the preset operation starting time threshold value of the current node, first early warning information is sent out through the current node;

8. The method of claim 1, further comprising:

9. A batch job monitoring device comprising:

the receiving module is used for receiving the job message from the upstream node in the batch job link;

an update module, configured to update a job packet of a current node in the batch job link using the job packet from the upstream node, where the current node is located downstream of the upstream node in the batch job link;

and the early warning module is used for sending out early warning information under the condition that the operation time of the current node meets the condition of a preset time threshold.

10. An electronic device, comprising:

one or more processors;

a storage device for storing one or more programs,

wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any of claims 1-8.

11. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 8.

12. A computer program product comprising a computer program which, when executed by a processor, implements a method according to any one of claims 1 to 8.