CN111737088B - Log acquisition method and device, electronic equipment and medium - Google Patents

Abstract

The disclosure provides a log obtaining method which is applied to a distributed batch processing platform, wherein the distributed batch processing platform comprises a job scheduling system and a job execution system, and the job execution system is arranged on a PAAS platform. The method comprises the steps of monitoring the running state of a first job from the job scheduling system, responding to the running state reaching a log acquisition condition corresponding to the first job, logging in the PAAS platform, and acquiring a first log generated by the job execution system executing the first job from the PAAS platform after logging in the PAAS platform. The disclosure also provides a log obtaining device, electronic equipment and a medium.

Description

Log acquisition method and device, electronic equipment and medium

Technical Field

The present disclosure relates to the field of computer technology, and more particularly, to a log obtaining method and apparatus, an electronic device, and a medium

Background

The distributed batch processing platform adopts a structure of separating job scheduling and job execution to simplify batch processing of a business system, for example, batch job scheduling can be carried out through a job scheduling system, and the job is executed through a job execution system. The job scheduling system may be, for example, a distributed batch framework, and may provide job scheduling services. The job execution system may be provided, for example, on a PAAS Platform-as-a-Service (Platform-as-Service) Platform, where servers or server clusters are dynamically virtualized to run batch jobs.

In the process of realizing the technical conception of the present invention, the inventors found that the prior art has at least the following drawbacks: in the related art, the distributed batch framework has limited capabilities in providing detailed logs, which results in that if it is desired to learn detailed job information through the logs, it is generally required to manually search the running state and scheduling information of the corresponding job from the distributed batch framework (e.g., find the IP address of the virtual server running the job and the corresponding job name, etc.), and then manually log into the PAAS platform to manually search the logs.

Disclosure of Invention

In view of this, the embodiments of the present disclosure provide a log obtaining method, a log obtaining device, an electronic device, and a medium that can automatically obtain a log in a distributed batch platform.

In one aspect of the disclosed embodiments, a log obtaining method is provided and applied to a distributed batch processing platform, wherein the distributed batch processing platform comprises a job scheduling system and a job execution system, and the job execution system is arranged on a PAAS platform. The method comprises the following steps: monitoring an operational status of a first job from the job scheduling system; responding to the running state reaching a log obtaining condition corresponding to the first job, and logging in the PAAS platform; and after logging in the PAAS platform, acquiring a first log generated by the first job executed by the job execution system from the PAAS platform.

According to an embodiment of the present disclosure, the monitoring the operation state of the first job from the job scheduling system includes: acquiring search information for searching the record information of the first job from the job scheduling system, wherein the record information of the first job is used for recording the running state of the first job and is information fed back to the job scheduling system by the job execution system; searching record information of the first job in the job scheduling system based on the search information; and monitoring the running state based on the recorded information of the first job.

According to an embodiment of the disclosure, the obtaining, from the PAAS platform, a first log generated by the job execution system executing the first job includes: determining a log searching path corresponding to the searching information, wherein the log searching path is used for searching the first log from the PAAS platform; and obtaining the first log from the PAAS platform based on the log lookup path.

According to an embodiment of the disclosure, the method further includes configuring the lookup information, and configuring a correspondence of the lookup information and the log lookup path.

According to an embodiment of the disclosure, the method further includes setting the log obtaining condition, where the log obtaining condition includes that the first job is in operation or an operation state of the first job is changed.

According to an embodiment of the disclosure, the method further comprises processing the first log, obtaining a second log, and storing the second log.

According to an embodiment of the present disclosure, the method further comprises displaying the second log.

In another aspect of the disclosed embodiments, a log obtaining device is provided and is disposed on a distributed batch processing platform, where the distributed batch processing platform includes a job scheduling system and a job execution system, and the job execution system is disposed on a PAAS platform. The device comprises a monitoring module, a login module and a log acquisition module. The monitoring module is used for monitoring the running state of the first job from the job scheduling system. And the login module is used for responding to the running state to reach a log acquisition condition corresponding to the first job and logging in the PAAS platform. The log obtaining module is used for obtaining a first log generated by the first job executed by the job execution system from the PAAS platform after logging in the PAAS platform.

According to an embodiment of the disclosure, the apparatus further comprises a configuration module. The configuration module is used for configuring the search information and the corresponding relation between the search information and the log search path.

According to an embodiment of the disclosure, the configuration module is further configured to set the log obtaining condition, where the log obtaining condition includes that the first job is in operation or an operation state of the first job changes.

According to an embodiment of the disclosure, the apparatus further comprises a log processing module. The log processing module is used for processing the first log to obtain a second log and storing the second log.

According to an embodiment of the disclosure, the log processing module is further configured to present the second log.

In another aspect of the disclosed embodiments, an electronic device is provided. The electronic device includes one or more memories, and one or more processors. The memory has stored thereon computer executable instructions. The processor executes the instructions to implement the method as described above.

In another aspect of the disclosed embodiments, a computer-readable storage medium is provided having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method as described above.

Another aspect of the disclosed embodiments provides a computer program comprising computer executable instructions which, when executed, are for implementing a method as described above.

One or more of the above embodiments have the following advantages or benefits: the operation state of the job can be automatically monitored, and when the log obtaining condition is reached, the job can be automatically logged in the PAAS cloud platform to obtain the corresponding log, so that the automatic log obtaining is realized, and the monitoring and processing of the job are more timely.

Drawings

The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments thereof with reference to the accompanying drawings in which:

FIG. 1 schematically illustrates a system architecture employing a log acquisition method and apparatus, and an electronic device, according to embodiments of the present disclosure;

FIG. 2 schematically illustrates a flow chart of a log acquisition method according to an embodiment of the present disclosure;

FIG. 3 schematically illustrates a flow chart of a log acquisition method according to another embodiment of the present disclosure;

FIG. 4 schematically illustrates a flow chart of a log acquisition method according to yet another embodiment of the present disclosure;

FIG. 5 schematically illustrates a block diagram of a log obtaining apparatus according to an embodiment of the disclosure;

FIG. 6 schematically illustrates a block diagram of a log obtaining apparatus according to another embodiment of the present disclosure;

fig. 7 schematically shows a block diagram of an information configuration module in the log obtaining apparatus shown in fig. 6;

FIG. 8 schematically illustrates a block diagram of a timing monitor module in the log acquisition apparatus shown in FIG. 6;

Fig. 9 schematically shows a block diagram of a log obtaining module in the log obtaining apparatus shown in fig. 6;

Fig. 10 schematically shows a block diagram of a log processing module in the log obtaining apparatus shown in fig. 6;

FIG. 11 schematically illustrates a block diagram of a foreground presentation module in the log obtaining apparatus shown in FIG. 6;

fig. 12 schematically illustrates an operation flowchart of a user using the log obtaining apparatus shown in fig. 6;

FIG. 13 schematically illustrates a flow chart of operation of the log obtaining apparatus shown in FIG. 6; and

Fig. 14 schematically illustrates a block diagram of a computer system suitable for implementing a log acquisition method according to an embodiment of the present disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is only exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and/or the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein should be construed to have meanings consistent with the context of the present specification and should not be construed in an idealized or overly formal manner.

Where a convention analogous to "at least one of A, B and C, etc." is used, in general such a convention should be interpreted in accordance with the meaning of one of skill in the art having generally understood the convention (e.g., "a system having at least one of A, B and C" would include, but not be limited to, systems having a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). Where a formulation similar to at least one of "A, B or C, etc." is used, in general such a formulation should be interpreted in accordance with the ordinary understanding of one skilled in the art (e.g. "a system with at least one of A, B or C" would include but not be limited to systems with a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).

In the related art, when a detailed log of a job is checked in a distributed batch processing platform, the PAAS platform is usually logged in manually, so that the operation is complex, and the monitoring of the job is delayed. Therefore, the embodiments of the present disclosure provide a log obtaining method, a log obtaining device, an electronic device, and a medium, which can automatically monitor an operation state of a job, and automatically log in a PAAS cloud platform to obtain a corresponding log when a log obtaining condition is reached, so as to achieve automatic log obtaining, and more timely monitor and process the job.

Specifically, the log obtaining method of the embodiment of the disclosure is applied to a distributed batch processing platform, wherein the distributed batch processing platform comprises a job scheduling system and a job execution system, and the job execution system is arranged on a PAAS platform. The method comprises the steps of firstly monitoring the running state of a first job from a job scheduling system, then logging in a PAAS platform in response to the running state reaching a log acquisition condition corresponding to the first job, and then acquiring a first log generated by a job execution system executing the first job from the PAAS platform after logging in the PAAS platform.

Fig. 1 schematically illustrates a system architecture 100 applying a log acquisition method and apparatus, and an electronic device according to an embodiment of the present disclosure.

As shown in fig. 1, a system architecture 100 according to this embodiment may include a distributed batch platform, and a log obtaining device 104 according to an embodiment of the present disclosure. Wherein the distributed batch platform comprises a job scheduling system 101 and a job execution system 102. The job scheduling system 101 is specifically a distributed batch framework DBF (distributed based framework). The job execution system 102 is disposed in the PAAS platform 103. Log obtaining device 104 may be connected to job scheduling system 101 and job execution system 102, respectively.

Job scheduling system 101 may provide a job scheduling service that schedules execution of batch jobs in job execution system 102. The job execution system 102 is disposed in the PAAS platform 103, and may specifically dynamically virtualize a plurality of virtual servers 21 to 24 in the PAAS platform 103 to execute corresponding jobs. For example, the plurality of virtual servers 21 to 24 may run the same application program to realize the batch job processing for a certain business. For example, for a credit card application, the plurality of virtual servers 21 to 24 may execute various types of operations such as a credit card operation, a credit card swiping operation, a payment operation, a preference recommendation operation, and a card opening operation, respectively.

Specifically, in the distributed batch platform, the job execution system 102 may register the address of the PAAS platform 103, the cluster name of the virtual server cluster running each batch job, the application program name, the server information of the execution job, and the like with the job scheduling system 101. Then, the job scheduling system 101 may perform job scheduling according to the registered information of the job execution system 102, and acquire and record the job execution situation fed back by the job execution system 102.

The log obtaining method of the embodiment of the present disclosure may be performed by the log obtaining apparatus 104. For example, the log obtaining device 104 may view or obtain information of a job execution condition of a job to be monitored (i.e., a first job in the present disclosure) from the job scheduling system 101, monitor an operation state of the first job, and automatically log in the PAAS platform 103 when the operation state of the first job reaches a log obtaining condition corresponding to the first job, and automatically obtain a log of the first job from the PAAS platform 103, so as to obtain detailed operation information of the first job.

The specific structure of the log obtaining device 104 may be, for example, the log obtaining device 500 described below with reference to fig. 5, or the log obtaining device 600 described below with reference to fig. 6, or the computer system 1400 described below with reference to fig. 14, or may be, for example, a computer-readable storage medium, or a program product, or the like.

The log obtaining device 104 may be independent of the job scheduling system 101 and the job execution system 102 as shown in fig. 1, or the log obtaining device 104 may be provided in the job scheduling system 101, for example, a program module that adds a corresponding function in the distributed batch framework DBF.

It should be noted that fig. 1 is only an example of a system architecture to which embodiments of the present disclosure may be applied to assist those skilled in the art in understanding the technical content of the present disclosure, but does not mean that embodiments of the present disclosure may not be used in other devices, systems, environments, or scenarios.

Fig. 2 schematically illustrates a flow chart of a log acquisition method according to an embodiment of the present disclosure.

As shown in fig. 2, the log obtaining method according to the embodiment may include operations S210 to S230. The log processing method is applied to a distributed batch processing platform, wherein the distributed batch processing platform comprises a job scheduling system 101 and a job execution system 102, and the job execution system 102 is arranged on a PAAS platform 103.

Specifically, first, in operation S210, the running state of the first job is monitored from the job scheduling system 101.

Then, in operation S220, in response to the running state reaching the log obtaining condition corresponding to the first job, the PAAS platform 103 is logged in. For example, the user login scenario may be simulated in the background based on the CACHE simulated login technique to log into the PAAS platform 103. The log acquisition condition may be a condition set in advance. For example, when the first job is a newly online job, the log obtaining condition may be set such that the first job is in operation, that is, the first job is triggered to obtain the log of the first job as long as the first job is in an operation state, so as to be convenient for comprehensively tracking detailed operation information of the first job; when the first job is a job that has been running for a long time, the log obtaining condition may be set such that the running state of the first job changes, including, for example, but not limited to, the first job starting running, the first job running ending, the first job running interruption, and the like.

Next, in operation S230, after logging in the PAAS platform 103, a first log generated by the job execution system 102 executing the first job is obtained from the PAAS platform 103. Therefore, the automatic acquisition of the log is realized, and the monitoring processing of the operation is more timely.

Fig. 3 schematically illustrates a flow chart of a log acquisition method according to another embodiment of the present disclosure.

As shown in fig. 3, the log obtaining method according to the embodiment of the present disclosure may include operations S211 to S213, S220, and S231 to S232. Operation S220 corresponds to the foregoing, and operations S211 to S213 are one embodiment of operation S210, and operations S231 to S232 are one embodiment of operation S230.

Specifically, first, in operation S211, search information for searching for the record information of the first job from the job scheduling system 101 is acquired. The recording information of the first job is information fed back to the job scheduling system 101 by the job execution system 102, and is used for recording the running state of the first job.

Then, in operation S212, the record information of the first job is searched in the job scheduling system 101 based on the search information.

The lookup information may include, for example, a server address of the distributed batch framework DBF, a name of a database associated with the first job in the distributed batch framework DBF, a name of a data table, a name of an application, and a job name of the first job.

The log obtaining device 104 may search the corresponding database, the data table, and the application program from the distributed batch frame DBF according to the search information, and refine and accurately find the record information of the first job layer by layer.

Next, in operation S213, the operation state of the first job is monitored based on the record information of the first job.

After that, in operation S220, in response to the running state of the first job reaching the log obtaining condition corresponding to the first job, the PAAS platform 103 is logged in.

Then, in operation S231, a log search path corresponding to the search information is determined, wherein the log search path is used to search the first log from the PAAS platform 103.

Finally, in operation S232, a first log is obtained from the PAAS platform 103 based on the log lookup path.

According to some embodiments of the present disclosure, the log obtaining method may further include configuration lookup information and a correspondence relationship between the configuration lookup information and the log lookup path before operation S211.

According to an embodiment of the present disclosure, the log lookup path may correspond to the aforementioned lookup information. After the log obtaining device 104 finds the record information of the first job in the distributed batch framework DBF through the search information, the log search path of the first job can be further obtained correspondingly according to the search information. The log lookup path may include, for example, an IP address of the PAAS platform 103, an address and cluster name of a virtual server cluster running the first job, a name of an application running the first job, server information for executing the first job, path information for a server executing the first job to write a log, and the like. Thus, when the log obtaining device 104 logs on to the PAAS platform 103, the first log can be searched according to the log searching path.

Fig. 4 schematically illustrates a flowchart of a log acquisition method according to yet another embodiment of the present disclosure.

As shown in fig. 4, the log obtaining method according to the embodiment may include operations S410 and S440 to S460 in addition to operations S210 to S230.

In operation S410, a log acquisition condition is set, wherein the log acquisition condition includes that the first job is in operation or that an operation state of the first job is changed.

Then, operations S210 to S230 are performed, thereby acquiring a first log.

Next, in operation S440, the first log is processed to obtain a second log. For example, each log in the first log is cleaned, identified, and/or processed in a uniform format to obtain the second log.

Thereafter, in operation S450, the second log is stored, thereby storing the second log locally. The problem that the log is cleaned and lost after the PAAS platform 103 is restarted is avoided. In case the PAAS platform 103 is restarted in case of e.g. an application upgrade, the PAAS platform 103 will re-deploy the virtual servers (clusters), and the log previously written in the PAAS platform 103 will be cleaned up. The method of the embodiment of the disclosure can effectively save the log of the first job in the distributed batch platform according to the requirement.

Thereafter, in operation S460, a second log is presented. And displaying the second log to the user according to the user requirement in a certain mode so as to help the user to read the detailed operation information of the first job.

Fig. 5 schematically illustrates a block diagram of a log obtaining apparatus 500 according to an embodiment of the disclosure.

As shown in fig. 5, the apparatus 500 may include a monitoring module 510, a login module 520, and a log acquisition module 530 according to an embodiment of the present disclosure. According to further embodiments of the present disclosure, the apparatus 500 may further include a configuration module 540, and/or a log processing module 550. The apparatus 500 may be disposed on a distributed batch platform comprising a job scheduling system 101 and a job execution system 102, wherein the job execution system 102 is disposed on a PAAS platform 103. The apparatus 500 may be used to implement the methods described with reference to fig. 2-4.

The monitoring module 510 may perform, for example, operation S210 for monitoring the running state of the first job from the job scheduling system 101.

The login module 520 may, for example, perform operation S220, for logging in the PAAS platform 103 in response to the running state reaching the log obtaining condition corresponding to the first job.

The log obtaining module 530 may, for example, perform operation S230, configured to obtain, from the PAAS platform 103 after logging in the PAAS platform 103, a first log generated by the job execution system 102 executing the first job.

The configuration module 540 is configured to configure the lookup information and a correspondence between the configuration lookup information and the log lookup path. Optionally, the configuration module 540 is further configured to set a log obtaining condition, where the log obtaining condition includes that the first job is in operation or that an operation state of the first job is changed (operation S410).

The log processing module 550 may perform operations S440 and S450, for example, to process the first log, obtain the second log, and store the second log. Optionally, the log processing module 550 may, for example, perform operation S460, and is further configured to present the second log.

The log obtaining apparatus and one application example of the log obtaining apparatus according to the embodiment of the present disclosure are described below with reference to examples of fig. 6 to 13, and it will be understood that fig. 6 to 13 are only schematic illustrations of application examples of the embodiment of the present disclosure, and do not constitute any limitation to the present disclosure.

Fig. 6 schematically illustrates a block diagram of a log obtaining apparatus 600 according to another embodiment of the present disclosure. The log obtaining device 600 is connected to one or more distributed batch frameworks DBFs, and may log into the PAAS platform 103 by means of log-in simulation.

As shown in fig. 6, the log obtaining apparatus 600 may include an information configuration module 1, a timing monitoring module 2, a log obtaining module 3, a log processing module 4, and a foreground presentation module 5.

The information configuration module 1 is used for storing configuration information of a user, and is a configuration database of the whole log obtaining device 600. The information configuration module 1 may configure, for example, search information of record information of the first job, a correspondence between the search information and a log search path, and/or a log acquisition condition according to a user operation. And providing behavior standards for the modules such as the timing monitoring module 2, the log acquisition module 3, the log processing module 4, the foreground display module 5 and the like through the configuration information.

The timing monitoring module 2 is configured to monitor a server database of the distributed batch framework DBF, and determine an operation state of a first job in the distributed batch framework DBF. The timing monitoring module 2 may determine whether the monitored first job reaches the log acquisition condition (for example, whether the first job is in operation or whether the operation state of the first job changes) based on the log acquisition condition configured in the information configuration module 1. And if the log obtaining condition is met, calling a log obtaining module 3 to carry out subsequent operation. Thereafter, the timing monitor module 2 continues to maintain the timing monitor state.

The log obtaining module 3 is connected with the timing monitoring module 2 and is used for logging in and obtaining a log file used for recording the first log on the PAAS platform 103 and transmitting the log file to the log processing module 4.

The log processing module 4 is connected with the log obtaining module 3 and is used for performing various processes on the first log. For example, each log in the log file is judged, cleaned, identified, and then the log is transferred to the foreground display module 5.

The foreground display module 5 is connected with the log processing module 4, and can be used for saving logs, is a log content database of the whole system, and can be used for displaying log data for users.

Fig. 7 schematically shows a block diagram of the information configuration module 1 in the log obtaining apparatus 600 shown in fig. 6.

As shown in fig. 7, the information configuration module 1 may include a timing monitor information unit 11, a log acquisition information unit 12, a log processing information unit 13, and a foreground presentation information unit 14.

The timing monitor information unit 11 stores timing monitor information configured by a user. The timing monitoring information includes search information for searching for recording information of the first job in the distributed batch framework DBF, log acquisition conditions of the first job, and the like. The lookup information may include, for example, a server address of the distributed batch framework DBF that schedules the first job, a name of a database and a name of a data table in which information of the first job is recorded, a name of an application program running the first job, a job name of the first job, and the like.

The log obtaining information unit 12 stores log obtaining information configured by a user, and stores, for example, a log search path corresponding to the aforementioned search information. The log lookup path may include an IP address of the PAAS platform 103, an address and cluster name of a virtual server cluster running the first job, a name of an application running the first job, server information for executing the first job, and path information for a server executing the first job to write a log.

The log processing information unit 13 stores log processing information configured by the user, including a cleaning standard, an identification standard, a format processing standard, and the like of the log. The user may set some or all of the contents in the above criteria for the log of the job to be monitored as needed.

The foreground display information unit 14 stores foreground display information configured by the user, including the name of the application program displayed, the job name, log entries, display effects, and the like.

Fig. 8 schematically shows a block diagram of the timing monitoring module 2 in the log obtaining apparatus 600 shown in fig. 6.

As shown in fig. 8, the log monitoring module 2 may include a timing monitoring operation unit 21, and a timing monitoring trigger unit 22. The log monitoring module 2 monitors the server database of the distributed job framework DBF, and determines whether or not the first job is running based on the running state of the first job or a change in the running state of the first job, thereby triggering the log acquisition module 3 at this timing.

The timing monitoring operation unit 21 reads the operation state of the first job from the distributed batch frame DBF frame using the timing script based on the timing monitoring information configured by the user.

And the timing monitoring triggering unit 22 is used for triggering successfully and calling the log acquisition module 3 when the running state of the first job meets the log acquisition condition.

Fig. 9 schematically shows a block diagram of the log obtaining module 3 in the log obtaining apparatus 600 shown in fig. 6.

As shown in fig. 9, the log obtaining module 3 includes a CACHE analog login unit 31 and a log copy unit 32. The log acquisition module 3 is used for automatically logging in the PAAS cloud platform and copying log files.

The CACHE simulated login unit 31 is based on the CACHE simulated login technology, and the background simulated user login scene is used for logging in the PAAS cloud platform, so that the login process does not need actual operation of the user.

The log copying unit 32 copies the log file to the local server under the log storage path of the virtual server cluster node running the first job in the PAAS platform 103 based on the log search path configured by the user.

Fig. 10 schematically shows a block diagram of the log processing module 4 in the log obtaining apparatus 600 shown in fig. 6.

As shown in fig. 10, the log processing module 4 may include a log judging unit 41, a log washing unit 42, and a log identifying unit 43. The log processing module 4 can use a logstack tool to judge, clean and identify the log.

The log judging unit 41 may read log files one by time, discard old logs before the triggering time of the current acquisition log, and reserve new logs after the triggering time of the current acquisition log. In this embodiment, in order to reduce the interaction cost with the PAAS platform 103, the log copying unit 32 takes the log in the PAAS platform 103 in full. In the processing, each log file is usually read from the head, and the previous log has no use value, so that the old log before the triggering time of the log can be abandoned according to time and the new log after the triggering time of the log can be reserved.

The log cleaning unit 42 classifies and summarizes routine detection logs in the system, establishes log rules, cleans out logs conforming to the rules based on the rules, and only retains job running logs. For example, some operating system class logs are purged.

The log identifying unit 43 identifies the relevant log by using keywords such as "ERROR", "WARNING", and the like, and may copy one additional log and send the same to the foreground display module 5 separately from the normal log.

Fig. 11 schematically shows a block diagram of the foreground presentation module 5 in the log obtaining apparatus 600 shown in fig. 6.

As shown in fig. 11, the foreground display module 5 may include a general log display unit 51, and a key log display unit 52. The foreground presentation module 5 uses elastic search and Kibana for log storage and foreground presentation.

The general log display unit 5 is used for displaying all log information.

The key log display unit 52 is used for displaying log information including keywords such as ERROR, WARNING, etc.

The background databases of the two units include all log contents acquired by the log acquisition unit 3.

Fig. 12 schematically shows an operation flowchart of a user using the log obtaining apparatus 600 shown in fig. 6.

As shown in fig. 12, the operation flow of the user using the log obtaining apparatus 600 shown in fig. 6 may include steps Y01 to Y03. Wherein the timing monitoring module 2 is running in the background throughout the operational flow.

In step Y01, the user uses the distributed batch framework DBF. In the process of using the distributed batch framework DBF by a user, after a new job is submitted to the distributed batch framework DBF based on user operation, the timing monitoring module 2 is triggered, the log obtaining module 3 and the log processing module 4 run immediately, and finally, the foreground display module 5 starts to run to provide a log storage function.

In step Y02, it is determined whether the log needs to be viewed.

In step Y03, when it is determined that the log needs to be viewed, the log is viewed. In this step, the foreground display module 5 starts to operate to provide a foreground display function.

And (3) the steps Y01 to Y03 are circulated at regular time, and the background is used for obtaining logs from the PAAS cloud platform at regular time until the operation is normally ended or interrupted.

Fig. 13 schematically shows a flowchart of the operation of the log obtaining apparatus 600 shown in fig. 6.

As shown in fig. 6, the operation flow of the log obtaining apparatus 600 may include steps Y11 to Y17.

In step Y11, the running state of the first job is monitored periodically from the distributed batch framework DBF.

In step Y12, it is determined whether the running state of the first job satisfies a preset log acquisition condition. Step Y11 and step Y12 are always in background operation. After the judgment is successful, a timing monitoring triggering step Y13 is executed.

In step Y13, the trigger is monitored at regular time to start the log obtaining module 3.

Step Y14, CACHE simulates login to PAAS platform 103.

Step Y15, performing log copying from the PAAS platform 103;

and step Y16, judging, cleaning and identifying the log.

And step Y17, storing the log. And then the whole process returns to the step Y11 and the step Y12, and the background operation is continued.

Any number of modules, sub-modules, units, sub-units, or at least some of the functionality of any number of the sub-units according to embodiments of the present disclosure may be implemented in one module. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented as split into multiple modules. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system-on-chip, a system-on-substrate, a system-on-package, an Application Specific Integrated Circuit (ASIC), or in any other reasonable manner of hardware or firmware that integrates or encapsulates the circuit, or in any one of or a suitable combination of three of software, hardware, and firmware. Or one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be at least partially implemented as computer program modules, which, when executed, may perform the corresponding functions.

For example, any of the monitoring module 510, login module 520, log acquisition module 530, configuration module 540, log processing module 550, information configuration module 1, timing monitoring module 2, log acquisition module 3, log processing module 4, foreground presentation module 5, timing monitoring information unit 11, log acquisition information unit 12, log processing information unit 13, foreground presentation information unit 14, timing monitoring running unit 21, timing monitoring trigger unit 22, CACHE analog login unit 31, log copy unit 32, log determination unit 41, log cleansing unit 42, log identification unit 43, general log presentation unit 51, and key log presentation unit 52 may be incorporated into one module for implementation, or any of the modules may be split into multiple modules. Or at least some of the functionality of one or more of the modules may be combined with, and implemented in, at least some of the functionality of other modules. According to embodiments of the present disclosure, at least one of the monitoring module 510, login module 520, log acquisition module 530, configuration module 540, log processing module 550, information configuration module 1, timing monitoring module 2, log acquisition module 3, log processing module 4, foreground presentation module 5, timing monitoring information unit 11, log acquisition information unit 12, log processing information unit 13, foreground presentation information unit 14, timing monitoring execution unit 21, timing monitoring trigger unit 22, CACHE analog login unit 31, log copy unit 32, log determination unit 41, log cleaning unit 42, log identification unit 43, ordinary log presentation unit 51, and critical log presentation unit 52 may be implemented at least in part as hardware circuitry, such as a Field Programmable Gate Array (FPGA), programmable Logic Array (PLA), system on a chip, system on a substrate, system on a package, application Specific Integrated Circuit (ASIC), or any other reasonable manner of hardware or firmware that may be integrated or packaged by circuitry, or any suitable combination of any of software, hardware, and firmware. Or at least one of the monitoring module 510, the login module 520, the log obtaining module 530, the configuration module 540, the log processing module 550, the information configuration module 1, the timing monitoring module 2, the log obtaining module 3, the log processing module 4, the foreground presentation module 5, the timing monitoring information unit 11, the log obtaining information unit 12, the log processing information unit 13, the foreground presentation information unit 14, the timing monitoring operation unit 21, the timing monitoring trigger unit 22, the CACHE analog login unit 31, the log copy unit 32, the log judgment unit 41, the log cleaning unit 42, the log identification unit 43, the general log presentation unit 51, and the key log presentation unit 52 may be at least partially implemented as a computer program module, which may perform a corresponding function when operated.

Fig. 14 schematically illustrates a block diagram of a computer system 1400 adapted to implement a log acquisition method according to an embodiment of the present disclosure. The computer system 1400 illustrated in fig. 14 is merely an example, and should not be taken as limiting the functionality and scope of use of the disclosed embodiments.

As shown in fig. 14, a computer system 1400 according to an embodiment of the present disclosure includes a processor 1401, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 1402 or a program loaded from a storage section 1408 into a Random Access Memory (RAM) 1403. The processor 1401 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or an associated chipset and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), or the like. The processor 1401 may also include on-board memory for caching purposes. The processor 1401 may include a single processing unit or a plurality of processing units for performing different actions of the method flows according to embodiments of the present disclosure.

In the RAM 1403, various programs and data necessary for the operation of the system 1400 are stored. The processor 1401, ROM 1402, and RAM 1403 are connected to each other through a bus 1404. The processor 1401 performs various operations of the method flow according to the embodiment of the present disclosure by executing programs in the ROM 1402 and/or the RAM 1403. Note that the program may be stored in one or more memories other than the ROM 1402 and the RAM 1403. The processor 1401 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in the one or more memories.

According to an embodiment of the present disclosure, the system 1400 may also include an input/output (I/O) interface 1405, the input/output (I/O) interface 1405 also being connected to the bus 1404. The system 1400 may also include one or more of the following components connected to the I/O interface 1405: an input section 1406 including a keyboard, a mouse, and the like; an output portion 1407 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like; a storage section 1408 including a hard disk or the like; and a communication section 1409 including a network interface card such as a LAN card, a modem, and the like. The communication section 1409 performs communication processing via a network such as the internet. The drive 1410 is also connected to the I/O interface 1405 as needed. Removable media 1411, such as magnetic disks, optical disks, magneto-optical disks, semiconductor memory, and the like, is installed as needed on drive 1410 so that a computer program read therefrom is installed as needed into storage portion 1408.

According to embodiments of the present disclosure, the method flow according to embodiments of the present disclosure may be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable storage medium, the computer program comprising program code for performing the method shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network via the communication portion 1409 and/or installed from the removable medium 1411. The above-described functions defined in the system of the embodiments of the present disclosure are performed when the computer program is executed by the processor 1401. The systems, devices, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.

The present disclosure also provides a computer-readable storage medium that may be embodied in the apparatus/device/system described in the above embodiments; or may exist alone without being assembled into the apparatus/device/system. The computer-readable storage medium carries one or more programs which, when executed, implement methods in accordance with embodiments of the present disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example, but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, the computer-readable storage medium may include ROM 1402 and/or RAM 1403 described above and/or one or more memories other than ROM 1402 and RAM 1403.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Those skilled in the art will appreciate that the features recited in the various embodiments of the disclosure and/or in the claims may be combined in various combinations and/or combinations, even if such combinations or combinations are not explicitly recited in the disclosure. In particular, the features recited in the various embodiments of the present disclosure and/or the claims may be variously combined and/or combined without departing from the spirit and teachings of the present disclosure. All such combinations and/or combinations fall within the scope of the present disclosure.

The embodiments of the present disclosure are described above. These examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the embodiments cannot be used advantageously in combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be made by those skilled in the art without departing from the scope of the disclosure, and such alternatives and modifications are intended to fall within the scope of the disclosure.

Claims (8)

1. The log obtaining method is applied to a distributed batch processing platform, wherein the distributed batch processing platform comprises a job scheduling system and a job execution system, the job execution system is arranged on a PAAS platform, and the method comprises the following steps:

monitoring an operational status of a first job from the job scheduling system, comprising:

Acquiring search information for searching record information of the first job from the job scheduling system;

Searching record information of the first job in the job scheduling system based on the search information, wherein the record information of the first job is used for recording the running state of the first job and is information fed back to the job scheduling system by the job execution system; and

Monitoring the running state based on the recorded information of the first job;

Determining a log searching path corresponding to the searching information; the search information comprises the name of an application program running the first job and the job name of the first job; the log searching path is from information related to the first job in the information registered by the job execution system to the job scheduling system, and the information registered by the job execution system to the job scheduling system comprises an IP address of the PAAS platform, a cluster name of a virtual server cluster running each batch job, an application program name and server information executing the job;

Responding to the running state to reach a log obtaining condition corresponding to the first job, and according to the IP address of the PAAS platform, simulating a user to log on the PAAS platform through a background login simulating technology; and

After logging in the PAAS platform, obtaining a first log generated by the job execution system executing the first job from the PAAS platform, wherein the first log comprises: and copying the log searched in the PAAS platform according to the log searching path to the local.

2. The method of claim 1, wherein the method further comprises:

configuring the search information; and

And configuring the corresponding relation between the searching information and the log searching path.

3. The method of claim 1, wherein the method further comprises:

Setting the log obtaining condition, wherein the log obtaining condition comprises that the first job is in operation or the operation state of the first job is changed.

4. The method of claim 1, wherein the method further comprises:

Processing the first log to obtain a second log; and

And storing the second log.

5. The method of claim 4, wherein the method further comprises:

The second log is shown.

6. A log obtaining apparatus applied to a distributed batch processing platform, the distributed batch processing platform comprising a job scheduling system and a job execution system, wherein the job execution system is disposed on a PAAS platform, the apparatus comprising:

The monitoring module is used for monitoring the running state of the first job from the job scheduling system and comprises the following components: firstly, acquiring search information for searching record information of the first job from the job scheduling system; then searching record information of the first job in the job scheduling system based on the search information, wherein the record information of the first job is used for recording the running state of the first job and is information fed back to the job scheduling system by the job execution system; then monitoring the running state based on the recorded information of the first job;

The log acquisition module is used for determining a log searching path corresponding to the searching information; the search information comprises the name of an application program running the first job and the job name of the first job; the log searching path is from information related to the first job in the information registered by the job execution system to the job scheduling system, and the information registered by the job execution system to the job scheduling system comprises an IP address of the PAAS platform, a cluster name of a virtual server cluster running each batch job, an application program name and server information executing the job;

The login module is used for responding to the running state to reach a log acquisition condition corresponding to the first operation, and simulating a user to login the PAAS platform through a background login simulating technology according to the IP address of the PAAS platform; and

The log obtaining module is further configured to obtain, from the PAAS platform after logging in the PAAS platform, a first log generated by the job execution system executing the first job, where the log obtaining module includes: and copying the log searched in the PAAS platform according to the log searching path to the local.

7. An electronic device, comprising:

One or more memories having computer-executable instructions stored thereon;

One or more processors executing the instructions to implement the method of any of claims 1-5.

8. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform the method according to any of claims 1-5.