JP5007247B2 - Job processing system and job management method - Google Patents

Description

  The present invention relates to information processing technology, and more particularly to a job management method for managing an abnormality that occurs during batch processing of jobs and a job processing system to which the method is applied.

  With the recent development of information processing technology and the enhancement of the network environment, various information has been transferred to the network, and it has been input to organizations that manage and manage individual data input to terminals such as companies and internal departments. Technology that strictly manages enormous amounts of data and the systems used is indispensable. Processing for managing data such as data backup and calculation of various numerical values, system maintenance, and the like are routine processing that is generally performed regularly such as daily or monthly. Therefore, it is often performed automatically at night by setting a plurality of jobs specified in advance to be processed in batches.

When batch processing jobs, the processing order of the jobs is determined in advance based on the processing capability, efficiency, dependency between jobs, priority order, and the like. By registering the processing contents of each job, that is, the job flow and its execution order in the system, basically, the desired processing is automatically completed at a desired time. As a result, it is possible to improve the efficiency of various processes while reducing labor costs (for example, Patent Document 1).
Japanese Patent Laid-Open No. 5-12037

  On the other hand, when a failure occurs in the middle of batch processing, recovery is often a difficult task. For example, when a job is stopped halfway, the cause is not only in the job itself, but also in a job executed before or a job executed before that. For example, when a plurality of jobs are executed in parallel, the number of jobs having a possible cause further increases. Job batch processing itself is premised on that it does not require a large number of personnel, but once a failure occurs, there are many parts where the cause investigation and recovery must be relied upon manually. This dilemma makes recovery more difficult.

  For this reason, in general, secure high-skilled personnel who can analyze the cause of failure, assign personnel with sufficient margins, and prepare multiple job flows for emergencies. In preparation for the occurrence of a failure. This results in increased costs such as labor costs, system development costs, and maintenance costs. This problem becomes more prominent as the number of processing restrictions increases, such as when batch processing must be completed by the business start time, and the system becomes larger. In addition, if a countermeasure is mistaken when a failure occurs, there is a risk that a secondary disaster such as a failure of a job that has been processed normally will occur. On the other hand, the burden on system developers is increasing in order to perform system design and job flow settings that do not cause such a situation.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide a job management technique capable of safely dealing with occurrence of a failure without increasing cost.

  One embodiment of the present invention relates to a job processing system. This job processing system is a job processing system for batch processing jobs, and each time a job is processed, either an execution status storage unit that accumulates and stores job execution status information, or a job being processed When an error occurs, refer to the execution status information, and the relative relationship between the processing time of the job in which the error has occurred and the processing time of other jobs that have been processed in the same batch process has been determined in the past performance. A failure cause detection unit that detects an abnormal event that deviates from the normal state that occurs with a probability exceeding the threshold as a cause candidate, and outputs information related to the cause candidate detected by the failure cause detection unit And an output unit.

  Here, the “execution status information” may be any information recorded in a general log, such as job processing start time, end time, resource access start time, end time, and error information.

  Another aspect of the present invention relates to a job management method. In this job management method, a step of acquiring used resource information in which a job batch-processed in a job processing system is associated with information on a resource used by each job, and a job execution status each time a job is processed Steps for storing and storing information, and when an error occurs in any of the jobs being processed, refer to the resource usage information and execution status information to determine whether there is an error in the resources used by the job, Whether the execution status of other jobs that use the same resource as the job and the relative relationship between the processing time of the job that has failed and the other job that uses the same resource as the job have changed from the normal state Failure cause candidates by checking at least one of the failure frequency in the past for the job in which the failure occurred Detecting, characterized in that it comprises the steps of: outputting information relating to candidates of the failure cause.

  It should be noted that any combination of the above-described constituent elements and a representation of the present invention converted between a method, an apparatus, a system, etc. are also effective as an aspect of the present invention.

  According to the present invention, a job can be processed without increasing the cost for preparing for the occurrence of a failure.

  FIG. 1 shows a configuration example of a system to which this embodiment can be applied. In FIG. 1, the job processing system 10 includes four servers: a first server 12, a second server 14, a third server 16, and a fourth server 18. The first server 12 is connected to the database 20. A user operates a terminal of each server to perform setting and registration to process a desired job at a desired time. The number of servers, databases, and database connection destinations are not limited to those shown in FIG. 1, and the present embodiment can be applied to any configuration as long as the system can process jobs. Further, a client terminal or the like may be connected to each server.

  Each of the first server 12, the second server 14, the third server 16, and the fourth server 18 generally includes one or more CPUs and memories, storage devices, input / output devices, display devices, or any combination thereof. As long as it is a typical information processing apparatus, the scale of a personal computer, general-purpose large computer, etc. is not limited. In the figure, as an example, the first server 12 has a hard disk 13 and the second server 14 has a hard disk 15. The first server 12 to the fourth server 18 are connected to the network 22 and can transmit and receive data to and from each other.

  The user performs job processing by setting the job flow, the processing order at the time of batch processing, the processing start time, etc. for any of the first server 12, the second server 14, the third server 16, and the fourth server 18. Cause the system 10 to process the job. One job may be processed by any one of the first server 12, the second server 14, the third server 16, and the fourth server 18, or may be processed by a plurality of servers. . In what order each server processes each job, and whether multiple jobs are processed in parallel, the available resources such as CPU processing capacity and network bandwidth, and the order of access to the database The user makes settings in view of the processing contents. These procedures can use general techniques used in job processing.

  In such a configuration, hardware such as a plurality of servers and databases and a plurality of jobs cooperate in a complex manner, and the process proceeds. At this time, if a job being processed by a certain server, for example, the first server 12, is stopped due to some trouble, the cause may be in the stopped job itself or in a completely different place. is there. For example, when erroneous data output by a job before the stopped job is read, when the job processing in the second server 14 has insufficient free space on the drive E of the hard disk 15 and writing cannot be performed, it is processed in parallel. There are various reasons for this, such as when a network connection time-out occurs due to a conflict with a job, or when a hardware failure occurs. Generally, it is necessary to manually verify these factors one by one, find the cause, overcome the problem, and re-execute the job processing.

  If it takes time to investigate the cause, it becomes impossible to finish all scheduled jobs at the scheduled time, and in some cases, the next day's business and work may be hindered. This creates greater risks as the system scales up. For example, if the first server 12 and the second server 14 are managed by different departments or provided in different places, the cause of abnormal termination of the job processed by the first server 12 is the second server 14. It is not easy to find it even if it is inside. If the problem in the second server 14 is overcome by the department that manages the problem while investigating the cause, the first server 12 becomes unaware of what caused the job to end abnormally. End up.

  As the amount of data to be processed increases and the scale of the system grows with the on-line and automation of various businesses that are increasingly accelerating, the above problems become more serious and burden on system developers and persons in charge of disabilities. Is increasing. Therefore, in this embodiment, a job related to the job in which the failure has occurred is automatically detected, and the cause of the failure occurrence is automatically narrowed down to efficiently support the recovery work. At this time, as the basis of relevance, (1) error information of a resource (hereinafter referred to as “used resource”) used by a job in which a failure occurred (hereinafter referred to as “failed job”), (2) failure The status of a job that uses the same resource as the job (hereinafter referred to as “same resource usage job”), (3) the relative status of the failed job and the same resource usage job, (4) the frequency of occurrence of the same failure, Focus on at least one of the four viewpoints.

  Although there is no direct connection between jobs from the viewpoint of processing contents, there are many cases where a relationship is accidentally generated from the viewpoint of failure occurrence. Such a fault relatedness of a job may occur due to various factors such as the job processing order and the amount of data to be processed, and is difficult to predict in advance. Even after a failure occurs, it is difficult to find the relevance only with the log of the target server or job. Therefore, in the present embodiment, when the failure occurs, the above four viewpoints are evaluated to link the jobs with each other as a medium to find the relationship on the failure. Then, the job and resource causing the failure are detected from the entire job processing system.

  FIG. 2 schematically shows a case where a relationship on a failure occurs due to a change in (3) the relative situation between the failed job and the resource use job among the above four viewpoints. In the figure, the horizontal axis represents the passage of time, and the time when three jobs, job A, job B, and job X are processed, is represented by respective rectangles. In this example, job A and job B are processed in this order, and job X is processed in parallel with these jobs. These jobs may be processed by the same server or may be processed by different servers. Consider a case where job X is abnormally terminated in an environment in which job batch processing is performed in such a processing order, that is, job X becomes a failed job. Normally, it is necessary to check the error log of the accessed resource, which is directly related to the job X, or the log of the job processed before the job X on the same server.

  On the other hand, there are many other events that can cause other than the factors that can be recognized as the cause of failure at first glance, and one of them is a relative situation with the resource use job. In the example of FIG. 2, in the “normal state” (upper part of the figure), the job X is being processed at the time of job A, that is, at time t1 earlier than the start time t2 of job B. However, in the “abnormal state” (the lower part of the figure) in which a failure has occurred in job X, the start time of job X is delayed due to some reason or the processing time of job X itself has increased. t3 is later than the start time t2 of job B. Here, the “normal state” is a state that occurs with the highest probability in the past operation results for a certain point of interest, and the “abnormal state” is the other state. In practice, a threshold value may be set for the occurrence probability, and a state higher than that may be referred to as a “normal state”, and a state lower than that may be referred to as an “abnormal state”. Further, two or more states may be included in the “normal state”.

  In such a situation, as shown in the figure, if job B and job X are jobs that access the same database 20, that is, if job B is a job using the same resource, job X ends abnormally. There can be. This is because the job B and job X access the database 20 at completely different times in the normal state, but the processing time of the job B and job X overlaps from time t2 to t3, so that the job X accesses the database 20 This is because it is possible that the job X cannot be performed or the job X is overwritten unscheduled and the job X cannot refer to regular data. In such a case, there is a faulty relationship between job X and job B, but there is no direct relationship, and the fact that the processing time of job X has shifted from the normal time is recognized. It is also difficult to search for the relationship manually.

  Furthermore, if job A, job B, and job X are processed on different servers or the management department is different, it is more difficult to conclude that the cause of abnormal termination of job X is the overlap of processing time with job B. It becomes. As the number of jobs processed in parallel increases, it becomes more difficult to grasp the range in which the change in the execution status affects, and the identification of the cause becomes nearly impossible. Therefore, in the present embodiment, when a failure occurs, the occurrence of an association that should be called accidental is detected from the above four viewpoints. Then, it extracts the combination of jobs and resources that may affect the failure (hereinafter also referred to as “cause candidates”), and shows the probability that it can be the cause of the failure to improve the efficiency of the cause investigation. And support recovery work. The “cause candidate” may be only a job or only a resource depending on the detected situation.

  FIG. 3 shows the configuration of the first server 12 in more detail. The second server 14, the third server 16, and the fourth server 18 may have the same configuration. The first server 12 includes, in addition to the hard disk 13, a job registration unit 32 in which a user registers a job flow, a use resource information acquisition unit 34 that acquires information on resources used by each job, a job flow and use resource information, and the like. Job information storage unit 42 to store, job processing unit 36 to process registered jobs, failure cause detection unit 38 to detect the cause when a failure occurs, various logs and past execution status in batch processing when a failure occurs An execution status storage unit 44 that stores information, a cause probability storage unit 46 that stores a probability that can be a cause of failure set for each situation, and an output unit 40 that outputs information on the detected cause of failure.

  In FIG. 3, each element described as a functional block for performing various processes can be configured by a CPU, a memory, and other LSIs in terms of hardware, and in terms of software, operations, file operations, and networks This is realized by a program for performing communication or the like. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any one.

  The job registration unit 32 is an interface for a user to register information necessary for job processing, such as job flow, that is, processing contents performed in each job and processing order of a plurality of jobs in batch processing. The job registration unit 32 may be a combination of a display device that displays a registration screen and an input device that performs input on the registration screen, such as a keyboard and a pointing device, and is a device used in a general system that processes jobs. Can be applied. The registered information is stored in the job information storage unit 42.

  The used resource information acquisition unit 34 extracts resources used by each job from the job flow accepted by the job registration unit 32 and creates a table of used resource information. The used resource information table is a table in which job names processed in batches in the job processing system 10 are associated with resources used by the jobs, servers, resource utilization rates, and the like. The job registration unit 32 stores the job flow data in the job information storage unit 42 in a predetermined format having fields such as a hard disk for input / output and a server to be accessed. The server name, used resources, etc. are extracted from the predetermined fields. The used resource information table created in this way is also stored in the job information storage unit 42. The resource used by the failed job, the resource usage job, the resource usage rate of the resource usage job, and the like are determined from the usage resource information and used for cause detection.

  The job processing unit 36 reads information such as the job flow registered by the user and the job processing order from the job information storage unit 42 and executes the information. For this, a method used in a general system for processing jobs can be applied. The job processing unit 36 writes and reads data to and from the hard disk 13 in accordance with job processing contents. The hard disk 13 can be accessed from other servers, that is, the second server 14, the third server 16, and the fourth server 18 via the network 22.

  The job processing unit 36 causes the execution status storage unit 44 to store execution status information such as a job log and an error log indicating the current status of execution during job processing. The execution status storage unit 44 accumulates and stores, as execution status information, logs recorded in batch processing when a failure occurs and information included in logs when batch processing has been performed in the past. By accumulating the execution status at the time of past batch processing, it is used to determine whether it is statistically “normal state” or “abnormal state”.

  In FIG. 2, the determination that the job X does not overlap the processing time of the job B in the “normal state” can be set in advance as the direct relationship between the jobs A and B and the job X is lower. Is no longer a thing. Therefore, in general, it cannot be determined that the overlap between the processing times of job X and job B is “abnormal state”, and the cause of the abnormal end of job X in FIG. Making it difficult. Therefore, after defining the "normal state" from the past data of the execution status, when a failure occurs, the cause of the failure is detected by extracting the event that was different from the "normal state" in the execution status of the batch processing at that time .

  The past execution status information may be in the form of a log that is recorded each time the job processing unit 36 performs batch processing, or only information used for cause detection may be extracted. For example, by storing the start time and end time of each job, it is possible to define what state the “normal state” in the environment as shown in FIG. 2 is. As the execution status information, the start time and end time of file transfer during processing of each job, the start time and end time of access to resources such as a hard disk, etc. may be stored, and the processing time may be further microscopically evaluated. . Other specific examples will be described later.

  The execution status storage unit 44 further stores a job that has become a failed job in the past and a cause candidate whose probability of causing a failure when the failure has occurred exceeds a predetermined threshold. This information is used to evaluate the probability that the cause candidate of the current failure can be caused from the frequency that caused the failure in the past. If the true cause can be determined after the failure has occurred, the user may register the cause in the execution status storage unit 44 to improve the accuracy of the occurrence frequency data.

  The cause probability storage unit 46 stores cause probability information in which the status of the cause candidates in the evaluation from the above four viewpoints is associated with a numerical value that expresses how much the status can cause a failure. The cause probability information is set in advance based on previous experience values, actual performance in the execution environment, test results, and the like. Or you may set theoretically in consideration of the range etc. which each situation influences. Specific examples will be described later.

  When a failure occurs, the failure cause detection unit 38 refers to utilization resource information and execution status information, and detects a cause candidate that is considered to be the cause of the failure from the above four viewpoints. And based on the cause probability information which the cause probability memory | storage part 46 memorize | stores, the probability that each may become a failure cause is acquired. A specific method will be described later.

  In this embodiment, since the cause of the failure is detected beyond the server, the usage resource information for all jobs batch-processed by the job processing system 10 regardless of which server is processing which job. The execution status information is shared among the first server 12, the second server 14, the third server 16, and the fourth server 18. Therefore, every time the information is updated in a certain server, the updated information is transmitted to another server to update the information held by each server. Alternatively, the same information is referred to by making the job information storage unit 42 and the execution status storage unit 44 of a certain server accessible from other servers.

  The output unit 40 may be a general output device such as a display device or a printer, and outputs a cause candidate detected by the failure cause detection unit 38 and a probability that it can be a cause of the failure.

  Next, the operation of the job processing system 10 having the above configuration will be described. FIG. 4 is a flowchart showing a processing procedure when a failure occurs. Although the figure shows a case where the probability of the cause of failure is obtained in a multifaceted manner by performing all the evaluations from the above four viewpoints, the evaluation may be performed by any one or a combination of a plurality of them. Good.

  First, a failure occurs during job processing (S32). Then, the failure cause detection unit 38 collects information on errors related to the used resources, and obtains a probability that can be a cause for each error content, thereby performing an evaluation based on errors in the used resources (S34). Specifically, referring to the use resource information in the job information storage unit 42, the use resource of the failed job is extracted. Furthermore, refer back to the error log from the time of failure occurrence and collect the error log of the resource being used. Next, the cause probability information stored in the cause probability storage unit 46 is referred to based on the error content described in the error log, and the probability that the error can be caused for each error is acquired.

  Next, the failure cause detection unit 38 performs evaluation based on the status of the resource use job (S36). Specifically, referring to the used resource information, the resource usage job and the usage rate for the used resource of the job are acquired. Then, the probability that the job can cause a failure for each usage rate is acquired from the cause probability information. Here, the resource usage rate of the same resource usage job is listed as a representative of the status of the same resource usage job. However, the present invention is not limited to this, and may be a parameter related to the status of processing of the same resource usage job and failure. That's fine. For example, the ratio of the processing time of the day to the average processing time of the resource use job may be used. In this case, the average processing time is stored in the execution status storage unit 44 for each job.

  Next, the failure cause detection unit 38 performs evaluation based on the relative status of the failed job and the resource use job (S38). Specifically, it is determined whether or not the relative relationship between the processing time of the failed job and the same resource use job is “abnormal”, and if it is “abnormal”, the same job for each status The probability that can cause a failure is acquired from the cause probability information. Detailed procedures will be described later. Further, the failure cause detection unit 38 refers to the execution status storage unit 44, and when the failure job has caused a failure in the past, based on the frequency at which each cause candidate is caused or the frequency estimated as the cause, By obtaining the probability that each can be a cause of failure, evaluation is performed based on the past occurrence frequency (S40).

  When the probabilities that each cause candidate can be caused by the evaluation from the above four viewpoints are acquired, the final probability that the cause candidate can cause the failure is calculated by multiplying the probabilities for each cause candidate (S42). ). However, the “probability” in the description so far and the process of multiplying it are examples, and the calculation method is not limited as long as it is a method of performing weighting by multilateral evaluation and totalizing the evaluation results. For example, “probability” may be replaced with “influence” and an index having a distribution centered at 1 may be used. Further, when the probability based on the evaluation from a certain viewpoint exceeds a predetermined threshold value, correction such as weighting may be performed so that the final probability of the cause candidate becomes high.

  Then, the output unit 40 outputs the cause candidates and the probabilities that the causes can be caused in association with each other (S44). The user can confirm the output result, investigate the cause of the failure by, for example, performing a verification operation in descending order of probability, and implement the failure response by overcoming the problem (S46).

  Next, the evaluation method based on the relative status of the failed job and the resource use job in S38 will be described in more detail. FIG. 5 shows a processing procedure in which the failure cause detection unit 38 detects and evaluates the cause of failure according to the relative processing status of the failure job and the resource use job. First, narrowing down jobs that are targets for determining “normal state” and “abnormal state” is performed (S50). Specifically, the resource use job obtained by referring to the use resource information in S36 of FIG.

  Next, information on the past execution status is read from the execution status storage unit 44, and “normal state” is defined for the relative relationship between the failed job and the target job (S52). Items that are subject to judgment on whether or not they are in the “normal state” (hereinafter referred to as “judgment items”) are preset events that can cause a failure and that can be detected by changes in the relative relationship. This is stored in the storage unit 44. For example, in order to detect the cause of the failure as shown in FIG. 2, “overlapping job processing time” may be included in the determination item. Then, it is defined as “normal state” that the processing times of job X and job B do not overlap.

  The overlap of job processing times can be calculated from the end time and start time of each job in the information on the effective status. Similarly, an overlap of file transfer time, an overlap of resource access time, and the like may be used as determination items. The overlap of these times may be determined as normal / abnormality based on two events indicating whether or not there is some overlap, or may be determined based on more detailed information such as how much overlap.

  In either case, an event whose occurrence probability is equal to or higher than a predetermined threshold is defined as a “normal state” as a result. For example, in the case of FIG. 2, if the probability that the processing times of job B and job X do not overlap at all is 80% or more, the state is defined as “normal state”. Alternatively, the degree of overlap may be defined by the standard deviation when expressed by variance. Further, an event having the highest occurrence probability may be set as a “normal state”. Any other statistical means may be used for the definition. Specific examples of failure cases other than the example shown in FIG. 2 and determination items set for detecting the cause will be described later.

  Next, the log of the current batch process stored in the execution status storage unit 44 is read, and among the determination items for which the definition of “normal state” is defined, an event that is out of the “normal state” is indicated as “abnormal state”. "Is extracted (S54). In the example of FIG. 2, an event that the processing times of job X and job B overlap can be detected as an “abnormal state”. Then, by referring to the probability that can cause the failure for each event stored in advance in the cause probability storage unit 46, the probability that the cause candidate that caused the event extracted in S54 can cause the failure is acquired ( S56).

  Next, the determination items used for the cause detection will be exemplified by giving failure cases other than the failure shown in FIG. FIG. 6 and FIG. 7 are diagrams for explaining failure cases whose causes can be detected by evaluating the relative status of jobs. The display method of FIG. 6 is the same as that of FIG. In the example of FIG. 6, job X is processed while job A is being processed in the “normal state”. Also, after the processing of job A is completed, the processing of job B is started. As shown in the figure, job X includes processing for deleting a file on a hard disk, and job B includes processing for creating a file and storing it on the same hard disk.

  In such an embodiment, if the target of deletion by Job X is a large file, if the job X does not delete the file before the job B file is stored, the capacity of the hard disk will be exceeded. It can be thought of. At this time, the file cannot be stored and job B may end abnormally. The “abnormal state” in FIG. 5 shows such a situation. For some reason, the processing start time of job X is delayed and shifted after the processing start time of job B, so that job B ends abnormally. A similar failure can occur when the start time of job B is advanced or the processing time of job X is prolonged. In such a case, in order to detect the cause of abnormal termination of job B, the determination item may include the context between jobs.

  Strictly speaking, more detailed criteria such as the relationship between the start times, the relationship between the start times and the end times, and the relationship between the times when accessing the hard disk may be used as the determination items. By setting the context between jobs as a determination item, it is defined as “normal state” that job X has ended before the processing start time of job B, based on past execution status information. The fact that the processing start time of job X is later than the processing start time of job B can be detected as “abnormal state” from the information on the effective status of the day. In the case of the failure as shown in the figure, the job X itself is not in a situation where an error occurs because the processing start time is delayed, so it is difficult to investigate the cause of the failure only by checking the error log. In the present embodiment, as described above, it is possible to easily find the possibility that the delay in starting the processing of job X is the cause of the failure.

  In the example of FIG. 7, job X and job Y are processed while job A is being processed in the “normal state”. Also, after the processing of job A is completed, the processing of job B is started. As shown in the figure, it is assumed that job B, job X, and job Y all include processing to transfer a file via the same network or bus. In such a mode, when most of the usable bandwidth of the network is used for file transfer by job X and job Y, even if job B tries to transfer the file, a transfer error occurs due to network congestion. B may end abnormally.

  The “abnormal state” in FIG. 7 shows the situation, and the processing start time of job X and job Y is delayed for some reason, and the file transfer of job X, job Y, and job B overlaps at the same time. Due to this, job B has ended abnormally. In such a case, in order to detect a delay in the processing start time of job X and job Y as the cause of abnormal termination of job B, as described above, overlap of processing time of each job or overlap of file transfer time Etc. may be included in the determination item, or the number of jobs with file transfer processed in parallel may be included in the determination item.

  A job that involves file transfer may be identified as another by having the user input it when registering the job flow and adding identification information to the job, or the resource usage information acquisition unit 34 may analyze the job flow. You may include in the above-mentioned utilization resource information. Alternatively, information on whether or not to perform file transfer may be acquired from a job log at the time of actual processing. Then, in the “normal state”, the overlapping of the processing time of the jobs accompanying the file transfer is judged from the disclosure time and the end time, and the maximum number of jobs processed simultaneously is defined. In the example of FIG. 7, it can be defined from the information on the execution status up to the previous day that jobs with up to two file transfers can be processed simultaneously. Based on the information on the effective status of the day, the processing start time of job B is before the processing end time of job X and job Y, and the job with three file transfers is processed at the same time. ”Can be detected.

  Even in the case of such a failure, the job X and the job Y themselves are not in a situation where an error occurs, so that it is difficult to recognize as the cause of the abnormal end of the job B, and it takes time to investigate the cause in a general method. Eventually, measures such as verifying the job process again are taken, but if the reproducibility is poor, the cause may not end up being investigated. If the “normal state” is defined as described above and the “abnormal state” is determined by comparing with the situation when a failure occurs, efficient cause investigation and recovery work can be performed. Although FIG. 7 shows a job for performing transfer processing using a network, similar detection is possible even if the network is replaced with a memory or an output device.

  Next, usage resource information will be described. FIG. 8 shows an example of the data structure of the usage resource information created by the usage resource information acquisition unit 34. The usage resource information table 100 includes a job name column 102, a usage server column 104, a usage resource column 106, a resource usage rate column 108, and a remarks column 110. The used resource information acquisition unit 34 adds entries to the used resource information table 100 each time a new job flow is registered.

  The job name column 102 describes the name of the job registered by the user. The use server column 104 describes the name of the server to which the resource used by each job belongs. In the use resource column 106, identification information of the resource to be used is described. The resource utilization rate column describes the utilization rate with respect to the resource capacity when each job uses the resource described in the utilization resource column 106. However, it is not necessary to set for processing that allows simultaneous access, such as database reference. The remarks column 110 describes an outline of processing when using the resource. As described above, the data described in each column can be basically acquired by the use resource information acquisition unit 34 by analyzing the job flow data. On the other hand, the resource utilization rate described in the resource utilization rate column 108 may be automatically calculated based on the difference between snapshots before and after job processing when the utilization resource information acquisition unit 34 actually processes the job. The user may register the actual value in the development machine or the like.

  In FIG. 8, for example, a job whose job name is “job B” accesses the database 20 of the first server 12 to refer to data (third line). Further, data is created and stored in the drive D of the hard disk 13 of the first server 12 (line 4). At this time, writing is performed using 10% of the drive D. Further, data transfer is performed using a LAN card (line 5). At this time, the utilization rate of the LAN card is 30%. The same applies to “job A”, “job X”, and “job Y”.

  The data structure of the used resource information is not limited to that shown in FIG. 8, and the processing contents, the configuration of the job processing system, the cases of faults up to that point, etc. In addition, a parameter that can be used for detecting the cause of the failure is appropriately selected. When such detailed records are included in the usage resource information, it is possible to detect the cause of the failure with higher accuracy.

  Next, the cause probability information referred to by the failure cause detection unit 38 when acquiring the probability that the cause candidate that generated the event detected as the “abnormal state” in S56 of FIG. 5 can cause the failure will be described. FIG. 9 shows an example of a cause probability table classified by execution situation that is set for the event in the “abnormal state” and stored in the cause probability storage unit 46. The execution probability cause probability table 120 includes a related resource column 122, an event column 124, and a probability column 126.

  Even if jobs have the same “abnormal state” in the context, the probabilities of possible causes vary depending on which resource the job is linked to the failure. Therefore, it is desirable to examine the degree of influence of the “abnormal condition” according to the resources used and set the probability more accurately. Therefore, in the related resource column 122, resources that are used in common by the faulty job and the target job are set, and the probability can be set depending on the resource. Further, an event that is expected to be extracted as an “abnormal state” is set in the event column 124, and a probability that each event can be a cause is set in the probability column 126.

  For example, it is assumed that the relationship between the processing time of a failed job and a target job using the same “database” as the job is “abnormal state”. In the “normal state”, it is defined that the target job is processed after the failed job, but the current day is processed at the same time. In this case, the probability that the event that they are processed simultaneously is the cause of the failure is set to “90%” as shown in the second row of the cause probability table 120 by execution status in the example of FIG. This situation is in the case of the failure shown in FIG. 2, and contention for access to the database occurs, so the probability that can be the cause is set high.

  On the other hand, even if the fact that the target job has not been processed is extracted as an “abnormal state”, access conflict does not occur, so it is unlikely that a failure will occur due to this. Therefore, the probability is set to “0%” as shown in the third line of the cause probability table 120 by execution status. Similarly, when the resource to be used in common is a hard disk or a LAN card, the probability is set for each event extracted in the same manner.

  However, when multiplying the probabilities evaluated from four viewpoints for each cause candidate, if the cause probability of “normal state” is set to 0%, the probability that the probability is finally calculated no matter how high the other viewpoints are Becomes 0%. In preparation for such a case, the probability of the cause may be set to “50%” or the like in the “normal state” as shown in the sixth row of the cause probability table 120 according to execution status in FIG. . The probability setting for the “normal state” may be different for each resource used in common. In this case, the failure cause detection unit 38 acquires the probability by referring to the execution probability cause probability table 120 even in the “normal state”.

  The cause probability information including the cause probability table 120 according to the execution situation is set in advance and stored in the cause probability storage unit 46 so that the determination result and the value of the probability conform to the actual operation. The user may be able to update.

  As described above, the state defined as the “normal state” is an event whose occurrence probability is equal to or higher than a predetermined threshold as a result. This threshold value may be set individually as appropriate in accordance with resources, servers, job processing contents, etc. that are commonly used. For example, when processing only jobs that are unlikely to cause a failure even if the access order changes slightly, extracting the slight change as "abnormal" makes it difficult to narrow down the causes. . In such a case, the threshold value is set low in order to widen the range of events determined as “normal state”. In this way, by adjusting the ratio of events to be extracted as “abnormal conditions” according to job processing contents and resources, extraction of meaningless events is prevented, and the efficiency of cause detection is increased.

For the same reason, the failure cause detection unit 38 extracts an event that has become an “abnormal state” for a certain determination item, acquires the probability that the event can be a cause from the execution probability cause probability table 120, and then determines the determination item. The acquired cause probability may be corrected according to the occurrence probability as the actual result of “normal state”. For example, when the threshold value of “normal state” is set to X%, “abnormal state” is actually extracted for a determination item whose occurrence probability of “normal state” is Y%, When the probability that can be the cause is Z%, the probability Z ′% that can be the cause after the correction is derived as follows.
Z ′ = Z + (1−Z) × | Y−X | / (1−X)

  As a result, judgment items whose actual occurrence probability Y% is larger than the threshold value X% and almost become “normal state” are high in anomaly of the event extracted as “abnormal state”. Increase the probability of being. For example, if the threshold value is 80%, the occurrence probability of the “normal state” is 84%, and the probability that can be the cause is 90%, the probability that can be the cause after the correction is 92%. By performing such correction, a more accurate cause probability can be shown. The above formula is an example. For example, during a test on a development machine or actual operation, the occurrence probability Y% of the “normal state” of the determination item and the probability that the “abnormal state” of the item has caused the failure It can be used when the distribution is known to be an approximate straight line. In addition to the formula used for correction, a function derived based on various probability distributions may be selected as appropriate.

  Next, a specific example of detecting the cause of failure from the error information of the resource used by the failed job among the above four viewpoints will be described. FIG. 10 shows an example of a probability table for each resource, which is cause probability information referred to by the failure cause detection unit 38 in S34 of FIG. The resource-specific probability table 130 includes a resource column 132, an error content column 134, and a probability column 136. If the error content described in the error content column 134 is recorded in the error log for the use resource described in the resource column 132, the probability described in the probability column 136 is the probability that the error is the cause. Actually, for one resource described in the resource column 132, a plurality of error contents may be set in the error content column 134, and a probability may be set in the probability column 136 for each error content.

  In the case of the failure shown in FIG. 2, it can be seen from the use resource information table 100 shown in FIG. 8 that the job X in which the failure has occurred uses the database. Next, looking at the error log of the database, if an error with the content “snapshot is too old” is recorded, the probability that the database can cause a failure is set to “95%”.

  Next, a specific example of detecting the cause of failure from the status of the resource use job itself among the above four viewpoints will be described. FIG. 11 shows an example of a probability table by resource utilization rate, which is cause probability information referred to by the failure cause detection unit 38 in S36 of FIG. The probability table 140 according to resource utilization rate includes a utilization rate column 142 and a probability column 144 for a resource free space. The utilization rate for the resource free space is obtained by converting the resource utilization rate of each job set in the resource utilization rate column 108 of the utilization resource information table 100 into the utilization rate for the resource free space immediately before the job processing. .

  If the resource is a hard disk or memory, the free space corresponds to the capacity of the storage area where data is not stored, and if it is a network, it corresponds to the unused bandwidth and is not occupied by other jobs or data. Is an area that can be used. The free space of the resource immediately before job processing can be acquired by a snapshot function provided by a general operation system.

  As the utilization rate for the free space increases, other jobs cannot use the resource and a failure is likely to occur. Therefore, the resource utilization rate probability table 140 is set as such. Further, the resource utilization rate probability table 140 may be prepared for each resource. In the case of the failure shown in FIG. 7, from the use resource information table 100 shown in FIG. 8, the jobs using the same LAN card as the job B, which is the failure job, are job X and job Y, and job X It can be seen that the utilization rates of job Y for the LAN card are “60%” and “30%”, respectively. Therefore, even if each job uses the LAN card alone, the probability that the job X and the LAN card are the cause and the probability that the job Y and the LAN card are the cause are both “95%”.

  The above is a case where the resource usage rate is adopted as the status of the resource usage job itself, but as described above, the status of the resource usage job that can be used in the present embodiment is not limited thereto. For example, attention may be paid to the processing time of the resource utilization job, or evaluation may be performed by combining them. FIG. 12 shows an example of a probability table for each processing time in which a probability is set for each ratio of the processing time of the day to the average processing time of the resource use job. The processing time probability table 150 includes a processing time / average processing time column 152 and a probability column 154 for the current day.

  Based on the ratio of the processing time of the day with respect to the past average processing time, the probability that the resource use job can cause a failure is acquired from the table. As the processing time of the resource use job is longer than usual, it is considered that some problem occurs in the resource or the troubled job is likely to be affected. Therefore, the processing time probability table 150 is set as such. The average processing time of each job is calculated and stored in the execution status storage unit 44 each time the job processing unit 36 processes the job. The failure cause detection unit 38 derives the processing time of the day from the job log of the day stored in the execution status storage unit 44 by the job processing unit 36.

  Next, a specific example of detecting the cause of failure from the frequency of occurrence of the same failure among the above four viewpoints will be described. FIG. 13 shows an example of a probability table classified by occurrence frequency, which is cause probability information referred to by the failure cause detection unit 38 in S40 of FIG. The occurrence frequency probability table 160 includes an occurrence frequency column 162 and a probability column 164. As described above, when a failure occurs, a cause candidate that is estimated to have a high probability or a job or resource that has been determined to be a real cause is accumulated and stored. When a failure occurs, the probability that the cause candidate can be the cause of the failure is acquired based on the frequency stored as the cause in the past for the failed job. As shown in FIG. 13, if the occurrence frequency is high, the probability that it is the cause is also set high. When a failure occurs, the failure cause detection unit 38 refers to the execution status storage unit 44 to obtain the occurrence frequency for each cause candidate, and then refers to the occurrence frequency probability table 160 to determine the probability for each cause candidate. get.

  FIG. 14 shows an example in which the cause of failure is evaluated from the four viewpoints described above, and the probability that each cause candidate can be the cause is finally calculated. The failure probability calculation table 170 includes a cause job column 172, a probability column 174, a cause resource column 176, a resource error probability column 178, a status of the resource use job status column 180, and a failure job / resource use job relative status. It includes a probability column 182 and a probability column 184 based on the occurrence frequency. The failure cause detection unit 38 includes a probability column 178 due to resource error in the failure probability calculation table 170, a probability column 180 based on the status of the resource use job, a probability column 182 based on the relative status of the failure job / resource use job, and a probability based on the occurrence frequency. Evaluation is performed from each viewpoint so as to fill the column 184, and the cause probability acquired for each cause candidate is acquired. Then, by multiplying these probabilities, the final probability is calculated as shown in the probability column 174. Here, as described above, a technique other than multiplication may be used.

  The data output by the output unit 40 may be only the cause job column 172, the probability column 174, and the cause resource column 176, or the failure probability calculation table 170 may be all output. In the former case, the failure probability calculation table 170 may not actually be created, and the final probability may be calculated by temporarily storing the probability based on the evaluation from each viewpoint in a register or the like. In the example shown in FIG. 14, the probability that a failure has occurred through “database” due to “job B” is “95%” due to a resource error, and the probability according to the status of the resource utilization job is “ It can be seen from each column that “95%”, the probability based on the relative status of the failed job / resource utilization job is “90%”, and the probability based on the occurrence frequency is “98%”. The final probability is “79.60%”, which is higher than other cause candidates. Therefore, the user can focus on the job B and the database, and verify the cause of the failure intensively.

  The above description is based on the assumption that no failure has occurred in the resource use job of the failed job. However, when the resource use job is acquired by referring to the use resource information in S36 of FIG. The presence / absence of a failure may be confirmed by referring to the log of that day. If a failure has occurred in two jobs that use a common resource, there is a high probability that some problem has occurred in that resource, causing a failure. Therefore, when a failure has occurred in the resource use job, weighting is performed so that the probability that the use resource is the cause of the failure is increased.

  When there are a plurality of the same resource use jobs for the same use resource, it is confirmed whether or not there is a failure in all the jobs. As the number of the same resource use jobs in which a failure has occurred increases, the cause probability of the resources shared by them is increased. Alternatively, when it is found that a failure has occurred in a predetermined number or more of the same resource use jobs, it may be considered that the resource that they use in common is the cause of the failure. For example, when a failure has occurred in three jobs, job B, job X, and job Z that use the drive D, it is considered that some problem has occurred in the drive D and the failure has occurred.

  In this case, without evaluating the cause of the subsequent failure, the resource is output to the output unit 40 and notified to the user, and then the processing is scheduled or is being processed. Processing of a job that uses the resource may be automatically stopped. This operation is performed by the failure cause detection unit 38 controlling the job processing unit 36. In this way, not only can the failure cause be detected and the probability output, but also one resource can be determined to be the cause of the failure, and other jobs can be stopped earlier than the user responds, It is possible to prevent a secondary disaster in which a failure propagates.

  As described above, according to the present embodiment described above, use resource information in which a job to be processed and a resource to be used are associated is prepared in advance. When a failure occurs, a failure cause candidate is detected based on the information, and the probability that each candidate can be the cause is output. The probability is acquired by evaluating at least one of the following four points: an error of the resource used, the status of the resource usage job, the relative status of the failed job and the resource usage job, and the frequency of occurrence of the failure. . As a result, it is possible to easily acquire a faulty connection between jobs that cannot be found at first glance, and to present information that is effective for handling a fault by the user.

  In addition, by sharing the usage resource information among the servers, the cause can be obtained for a job being processed in a server with a different management department or installed in another location. Further, when all the evaluations from the four viewpoints are performed, the accuracy of the probability is improved by the multi-faceted evaluation. By displaying the probability, the user can easily prioritize the verification work, and the efficiency of the recovery work increases.

  When evaluating the relative status between the faulty job and the resource use job, the execution status is accumulated and stored, and the “normal status” is defined based on this. By determining the “abnormal state” with respect to the “normal state”, an event that is not recognized as “abnormal” from the log or the like can be detected as the cause of the failure. A job that is a target for determining whether or not it is in the “normal state” is the same resource use job that is obtained by referring to the use resource information, whereby efficient cause candidates can be narrowed down.

  Also, if a failure occurs in multiple jobs that use a common resource, it is determined that the probability that the resource is the cause of the failure is extremely high, and unprocessed jobs that use that resource are stopped. To do. As a result, it is possible to promptly take urgent responses to obvious obstacles without helping humans, and prevent the spread of damage.

  Since this embodiment can be implemented without introducing new hardware, it can be easily applied to an existing system at low cost. In addition, because it is possible to identify in detail the factors that can cause the failure, support is provided for system development, such as extracting improvement points by introducing it to test operations on development machines as well as operation machines. You can also.

  The present invention has been described based on the embodiments. This embodiment is an exemplification, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are also within the scope of the present invention. is there.

It is a figure which shows the structural example of the system which can apply this Embodiment. It is a figure which shows typically the case where the relationship on a failure generate | occur | produced with the relative condition of a failure job and the same resource utilization job. It is a figure which shows the structure of the 1st server in this Embodiment in detail. It is a flowchart which shows the process sequence at the time of the failure generation in this Embodiment. 6 is a flowchart illustrating a processing procedure in which a failure cause detection unit detects and evaluates a failure cause based on a relative situation between the failure job and the resource use job in the present embodiment. It is a figure explaining the failure example which can detect a cause by evaluation of the relative relationship of the job of this Embodiment. It is a figure explaining the failure example which can detect a cause by evaluation of the relative relationship of the job of this Embodiment. It is a figure which shows the example of the data structure of the utilization resource information which a utilization resource information acquisition part produces in this Embodiment. It is a figure which shows the example of the cause probability table classified by execution condition set with respect to the event which is an "abnormal state" in this Embodiment. It is a figure which shows the example of the probability table classified by resource which is cause probability information which a failure cause detection part refers in S34 of FIG. It is a figure which shows the example of the probability table classified by resource utilization rate which is the cause probability information which a failure cause detection part refers in S36 of FIG. It is a figure which shows the example of the probability table classified by processing time which set the probability for every ratio of the processing time of the day with respect to the average processing time of the same resource utilization job in this Embodiment. It is a figure which shows the example of the probability table classified by occurrence frequency which is the cause probability information which the failure cause detection part 38 refers in S40 of FIG. It is a figure which shows the example which evaluated the cause of a failure from four viewpoints in this Embodiment, and calculated the probability which each job and resource can cause finally.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 Job processing system, 12 1st server, 13 Hard disk, 14 2nd server, 20 Database, 32 Job registration part, 34 Use resource information acquisition part, 36 Job processing part, 38 Fault cause detection part, 40 Output part, 42 Job Information storage unit, 44 execution status storage unit, 46 cause probability storage unit.

Claims (9)

A job processing system for batch processing jobs,
An execution status storage unit that accumulates and stores job execution status information each time a job is processed,
When an error occurs in one of the jobs being processed, refer to the execution status information, and the degree of overlap in processing time between the job in which the error occurred and other jobs that were being processed in the same batch process Is a failure cause detection unit that detects, as a cause candidate, an event that is in an abnormal state that deviates from a normal state that occurs at a probability that exceeds a predetermined threshold in the past performance,
An output unit that outputs information related to a cause candidate detected by the failure cause detection unit. A cause probability storage unit for storing a cause probability table for each execution situation in which an event predicted to be extracted as an abnormal state is associated with a set value of a probability that the event may cause a failure;
The failure cause detection unit acquires a probability that an event detected as the cause candidate with reference to the execution situation-specific cause probability table may be a failure cause,
The job processing system according to claim 1, wherein the output unit outputs the event detected as the cause candidate and the probability that the event can be a cause of the failure in association with each other. A job information storage unit that stores use resource information in which a job processed in the job processing system is associated with a resource used by the job;
The failure cause detection unit identifies an error among the events detected as the cause candidate when an error is recorded in the log of the resource used by the job in which the failure has occurred, identified with reference to the use resource information. 3. The job processing system according to claim 2, wherein weighting is performed so as to increase a probability that an event related to a recorded resource may cause a failure. A job information storage unit that stores use resource information in which a job processed in the job processing system is associated with a resource used by the job and a use rate when the job uses the resource;
The failure cause detection unit refers to the used resource information, identifies the resource usage rate of another job that uses the same resource as the job in which the failure has occurred, and batches when the failure has occurred In the processing, weighting is given to the probability that an event related to the other job among the events detected as the cause candidate can be a cause of failure according to the utilization rate of the other job with respect to the free space of the resource. The job processing system according to claim 2, wherein the job processing system is performed. The execution status storage unit further stores a job in which a failure has occurred in the past in association with the cause of the failure,
The failure cause detection unit weights a probability that an event detected as the cause candidate can become a cause of failure according to a frequency of occurrence of a failure in the past for a job in which a failure has occurred. The job processing system according to 2.   The failure cause detection unit corrects a probability that an event detected as the cause candidate can be a cause of failure according to a difference between a probability of occurrence of a normal state in a past record and the threshold value. The job processing system according to claim 2.   The failure cause detection unit, when a failure has occurred in a job that uses the same resource as the job in which the failure has occurred, among the events detected as the cause candidate, the event related to the resource is the cause of the failure The job processing system according to claim 4, wherein weighting is performed so as to increase a probability of being possible.   When a failure has occurred in a predetermined number or more of jobs using the same resource as the job in which the failure has occurred, the failure cause detection unit estimates the resource as a failure cause, 8. The job processing system according to claim 4, wherein control is performed so as to stop a job being processed and an unprocessed job. Acquiring resource usage information that associates jobs to be batch processed in the job processing system with information relating to resources used by each job;
Each time a job is processed, a step of accumulating and storing job execution status information;
When an error occurs in one of the jobs being processed, the use resource information and the execution status information are referenced to check whether there is an error in the resource used by the job and use the same resource as the job. The execution status of other jobs, whether there is a change from the normal state of the degree of overlap of processing time of other jobs that use the same resource as the job in which the failure occurred, and the job that has failed Detecting at least one of the frequency of causes of failures in the past and detecting candidate causes of failures,
Outputting information relating to the failure cause candidate;
Including a job management method.

Images