CN111782621A

CN111782621A - Service application log processing method and device

Info

Publication number: CN111782621A
Application number: CN202010617619.2A
Authority: CN
Inventors: 刘晓辉; 周凯洋; 李婷; 岳云霞; 于泽
Original assignee: China Travelsky Holding Co
Current assignee: China Travelsky Holding Co
Priority date: 2020-06-30
Filing date: 2020-06-30
Publication date: 2020-10-16
Anticipated expiration: 2040-06-30
Also published as: CN111782621B

Abstract

The invention provides a method and a device for processing service application logs, wherein the method comprises the following steps: acquiring a service application log to be processed according to a first preset time interval; carrying out fragmentation processing on the service application log to obtain a plurality of fragment data; screening a plurality of fragment data based on a preset log type to obtain a fragment log with a determined log type; and processing the fragment logs of the corresponding types according to the processing modes of the fragment logs of different log types to obtain a processing result. In the embodiment of the invention, the log type of each fragmented log is determined through the preset log type, and the fragmented logs of different log types are processed to determine the fragmented logs capable of constructing the loopback data in the fragmented logs of a plurality of log types, so that the completeness and the accuracy of the log data in the process of processing the service application logs are ensured.

Description

Service application log processing method and device

Technical Field

The invention relates to the technical field of big data, in particular to a method and a device for processing service application logs.

Background

With the rapid development of civil aviation informatization, an application program running on a large-scale host system generates a large number of logs every day, the logs record the running state of the host system, and as the logs are interactive logs and in order to know the running state of the host system, a plurality of logs need to be subjected to context comparison when being processed to form a data structure with complete semantics.

Currently, logs are acquired from a message queue one by one based on the time sequence of log generation, and a single log is processed. In the process of processing the log context, the service application log cannot be directly obtained from the large-scale host system, and the interactive log can only be obtained through the peripheral equipment, so that the sending request and the reply request of the service application exist in different logs, and a single log only contains the sending request or the reply request. By processing the log in the above way, the situation that the context of the log is not matched is easy to occur, so that the problems of incomplete data and inaccuracy exist in the processing process of the log.

Disclosure of Invention

In view of this, embodiments of the present invention provide a method and an apparatus for processing a service application log, so as to solve the problems of incomplete data and inaccuracy in the log processing process due to mismatch of log contexts in the prior art.

In order to achieve the above purpose, the embodiments of the present invention provide the following technical solutions:

the first aspect of the embodiment of the invention discloses a method for processing service application logs, which comprises the following steps:

acquiring a service application log to be processed according to a first preset time interval;

carrying out fragmentation processing on the service application log to obtain a plurality of fragment data;

screening the plurality of fragment data based on a preset log type to obtain a fragment log with a determined log type;

and processing the fragment logs of the corresponding types according to the processing modes of the fragment logs of different log types to obtain a processing result.

Optionally, the performing fragmentation processing on the application log to obtain a plurality of fragmentation data includes:

the business application logs to be processed are subjected to fragmentation processing to obtain a plurality of fragmentation logs and log offsets corresponding to the fragmentation logs, each fragmentation log and the log offsets corresponding to the fragmentation logs form fragmentation data, each fragmentation log comprises log data and redundant data, and the redundant data is used for storing reply messages which are not contained in the log data;

and temporarily storing the fragment data into an internal cache, and storing the log offset into a message management queue.

Optionally, the screening the plurality of fragmented data based on a preset log type to obtain a fragmented log with a determined log type includes:

for each piece of fragmented data, reading a fragmented log corresponding to the log offset in an internal cache according to the log offset in the fragmented data;

filtering invalid fragment logs in the plurality of fragment logs based on a pre-stored server name and a monitoring index to obtain valid fragment logs;

and screening the effective fragmented logs based on a preset log type to obtain fragmented logs with the log type being the preset log type, wherein the preset log type comprises an IN-system component output interaction DAUDIT _ OUT type, an inter-system interaction input STAT _ IN type and an inter-system interaction output STAT _ OUT type.

Optionally, the processing the fragmented logs of the corresponding types according to the processing modes of the fragmented logs of different log types to obtain a processing result includes:

if only one or two types of fragment logs of a DAUDIT _ OUT type, a STAT _ IN type and a STAT _ OUT type exist IN the fragment logs obtained by screening, discarding the service application logs;

if the fragmentation logs of the DAUDIT _ OUT type, the STAT _ IN type and the STAT _ OUT type exist IN the fragmentation logs obtained through screening, searching the transaction ID of the fragmentation log of the DAUDIT _ OUT type aiming at the fragmentation log of each DAUDIT _ OUT type;

searching whether all effective fragment logs contain STAT _ IN type fragment logs and STAT _ OUT type fragment logs containing the transaction ID within preset time according to the transaction ID;

if the STAT _ OUT type fragmentation log containing the transaction ID does not exist, discarding the DAUDIT _ OUT type fragmentation log;

if the STAT _ IN type fragmentation log containing the transaction ID does not exist, determining to record a log record corresponding to the DAUDIT _ OUT type fragmentation log as an overtime record;

if the transaction IDs exist, the fragment logs of the DAUDIT _ OUT type, the fragment logs of the STAT _ IN type and the fragment logs of the STAT _ OUT type which contain the same transaction ID form initial loop data;

and processing the initial loop data to obtain a processing result.

Optionally, the processing the initial loopback data to obtain a processing result includes:

determining request information in the initial loopback data, wherein the request information is used for indicating that a user carries out input operation and outputs replied request information in the initial loopback data, and the request information comprises a synchronous request or an asynchronous request;

if the request is an asynchronous request, searching whether all the fragment logs contain the fragment logs of other STAT _ IN types which are the same as the user ID or not within preset time according to the user ID IN the STAT _ IN type fragment logs;

if yes, determining whether asynchronous identification exists IN the other STAT _ IN type fragment logs;

if the asynchronous identifier exists, resetting the preset time, and searching whether the successful identifier or the failed identifier exists IN the other STAT _ IN type fragment logs within the reset preset time;

if the success identification or the failure identification exists, recording the response time of finding the STAT _ IN type fragment log containing the success identification or the failure identification;

and if the success identifier or the failure identifier does not exist, recording the log record corresponding to the STAT _ IN type fragment log as an overtime record.

Optionally, the method further includes:

summarizing the processing results, and calculating an early warning value based on the summarized processing results;

judging whether the early warning value is equal to or greater than a set early warning value; if yes, alarming.

Optionally, the method further includes:

and acquiring and displaying a processing result of the service application log to be processed according to a second preset time interval.

The second aspect of the embodiments of the present invention discloses a service application log processing apparatus, where the apparatus includes:

the acquisition module is used for acquiring a service application log to be processed according to a first preset time interval;

the fragmentation module is used for carrying out fragmentation processing on the application log to obtain a plurality of fragmentation data;

the screening module is used for screening the plurality of fragmented data based on the log type to obtain a fragmented log with the determined log type;

and the processing module is used for processing the fragmentation logs of the corresponding types according to the processing modes of the fragmentation logs of different log types to obtain processing results.

The third aspect of the embodiments of the present invention discloses an electronic device, where the electronic device is configured to run a program, and when the program runs, the method for processing a service application log according to the first aspect of the embodiments of the present invention is executed.

A fourth aspect of the embodiments of the present invention discloses a computer storage medium, where the storage medium includes a storage program, and when the program runs, a device on which the storage medium is located is controlled to execute the service application log processing method according to the first aspect of the embodiments of the present invention.

Based on the method and the device for processing the service application log provided by the embodiment of the invention, the method comprises the following steps: acquiring a service application log to be processed according to a first preset time interval; carrying out fragmentation processing on the service application log to obtain a plurality of fragment data; screening a plurality of fragment data based on a preset log type to obtain a fragment log with a determined log type; and processing the fragment logs of the corresponding types according to the processing modes of the fragment logs of different log types to obtain a processing result. In the embodiment of the invention, the log type of each fragmented log is determined through the preset log type, and the fragmented logs of different log types are processed to determine the fragmented logs capable of constructing the loopback data in the fragmented logs of a plurality of log types, so that the completeness and the accuracy of the log data in the process of processing the service application logs are ensured.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.

Fig. 1 is a schematic architecture diagram of a service application log processing system according to an embodiment of the present invention;

fig. 2 is a schematic flowchart of a method for processing a service application log according to an embodiment of the present invention;

fig. 3 is a schematic flowchart of a process of performing fragmentation processing on a service application log according to an embodiment of the present invention;

fig. 4 is a schematic flowchart of processing fragmented logs of different log types according to an embodiment of the present invention;

fig. 5 is a schematic flowchart of processing fragmented logs of different log types according to an embodiment of the present invention;

fig. 6 is a schematic flowchart of another service application log processing method according to an embodiment of the present invention;

fig. 7 is a schematic flowchart of another service application log processing method according to an embodiment of the present invention;

fig. 8 is a schematic structural diagram of a service application log processing apparatus according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

In this application, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

In the embodiment of the invention, the log type of each fragmented log is determined through the preset log type, and the fragmented logs of different log types are processed to determine the fragmented logs capable of constructing the loopback data in the fragmented logs of a plurality of log types, so that the completeness and the accuracy of the log data in the process of processing the service application logs are ensured.

Fig. 1 is a schematic structural diagram of a service application log processing system according to an embodiment of the present invention. The service application log processing system comprises: a log collection component 11 and a log analysis component 12.

The log analysis component 12 includes a message management mechanism 121 and a temporary storage mechanism 122.

The architecture based on the system realizes the processing process aiming at the service application log, and comprises the following steps:

the log collection component 11 collects logs on each service application according to a first preset time interval by using a collection technology deployed on the application server, and stores the collected service application logs into a message queue Kafka of the log collection component 11.

Analyzing and processing the service application log by using a distributed system Hadoop deployed on the log analysis component 12 or a Spark environment of a CDH platform, specifically: the message management mechanism 121 obtains the service application log stored in the message queue Kafka in the log collection component 11. Performing fragmentation processing to obtain a plurality of fragment data, and temporarily storing the fragment data into an internal cache, i.e., a temporary storage mechanism 122; screening a plurality of fragmented data according to a preset log type, so that the log type of each fragmented log can be determined; and finally, sequentially processing the fragment logs of the corresponding types according to the processing modes of the fragment logs of different log types to obtain processing results, and temporarily storing the processing results in the temporary storage mechanism 122.

With continued reference to fig. 1, the service application log processing system is further provided with a monitoring alarm component 13 and an output display component 14.

The monitoring alarm component 13 monitors the processing results in the temporary storage mechanism 122 in real time, performs summary calculation on the processing results, and gives an alarm when it is determined that the calculated early warning value is equal to or greater than the set early warning value.

The output display component 14 retrieves the processing results in the temporary storage mechanism 122 and refreshes the display at a second preset time interval.

In the embodiment of the invention, a log acquisition component acquires and stores a service application log to be processed according to a first preset time interval; a message management mechanism in the log analysis component performs fragmentation processing on the service application log to obtain a plurality of fragment data, and the fragment data is stored in a temporary storage mechanism of the log analysis component; a message management mechanism in the log analysis component screens a plurality of fragment data based on a preset log type to obtain a fragment log with a determined log type; and processing the fragment logs of the corresponding types according to the processing modes of the fragment logs of different log types to obtain a processing result. In the embodiment of the invention, the log type of each fragmented log is determined through the preset log type, and the fragmented logs of different log types are processed to determine the fragmented logs capable of constructing the loopback data in the fragmented logs of a plurality of log types, so that the completeness and the accuracy of the log data in the process of processing the service application logs are ensured. Furthermore, a monitoring alarm component and an output display component are additionally arranged in the service application log processing system, so that a user can better know the processing condition of the log, and the user can provide certain processing measures conveniently.

Based on the service application log processing system, referring to fig. 2, a flow diagram of a service application log processing method provided by the embodiment of the present invention is shown, where the method includes:

step S201: and acquiring a service application log to be processed according to a first preset time interval.

In the process of implementing the step S201 specifically, the log collecting component reads the to-be-processed service application log located at the specified position of the application server according to the first preset time interval by using the distributed data collecting agent technology deployed on the application server, and temporarily stores the collected to-be-processed service application log into the message queue Kafka of the log collecting component.

It should be noted that the first preset time interval is set through a plurality of experiments, and may also be set according to experience of a technician, for example: may be set to 2 minutes.

Optionally, the application server is a peripheral device of the mainframe system, and the designated location of the application server records input and output log data of the mainframe system.

Step S202: and carrying out fragmentation processing on the service application log to obtain a plurality of fragment data.

In the process of implementing step S202, the following steps are included:

step S11: and carrying out fragmentation processing on the service application log to obtain a plurality of fragmentation logs and log offsets corresponding to the fragmentation logs.

In step S11, each fragmentation log and the log offset corresponding to the fragmentation log form a fragmentation data, each fragmentation log includes log data and redundant data, and the redundant data is used to store a reply packet that is not included in the log data.

Note that the log offset includes a log start offset and a log end offset. If the fragmentation log corresponding to the log offset is not the last fragmentation log, the log ending offset corresponding to the fragmentation log is the starting offset of the next fragmentation log.

In the process of specifically implementing step S11, the message management mechanism obtains the service application logs to be processed that are temporarily stored in the message queue Kafka, and segments the service application logs to be processed according to the preset number of segments to obtain the segment logs corresponding to the preset number of segments and the log start offset and the log end offset of the corresponding segment logs.

It should be noted that the preset number of slices is set through a plurality of experiments, and may also be set according to the experience of the technician, for example, may be set to 10.

The log offset is the distance between the actual address of the fragmented log and the segment address of the segment in which the fragmented log is located.

Step S12: and temporarily storing the fragment data into an internal cache, and storing the log offset into a message management queue.

In the process of implementing step S12 specifically, the fragment data is temporarily stored in an internal buffer of the Spark cluster, that is, the temporary storage mechanism, and the log offset is stored in a message management queue inside Kafka.

Alternatively, the log offset may be stored in a message management queue inside Kafka, a distributed application coordination service Zookeeper, or a high-performance key-value database Redis.

The log offset stored in the message management queue Kafka may be managed by a message management authority, such as: and saving the log ending offset corresponding to the read fragment log into a message management queue Kafka to serve as the log starting offset of the next fragment log reading.

Optionally, in the process of fragmenting the service application log to be processed, a message management mechanism in the log analysis component determines whether there is a request message in the log data in each fragment log and the redundant data corresponding to the log data, and a reply message corresponding to the request message, and if there is a request message, discards the redundant data in the fragment log.

Step S203: and screening a plurality of fragment data based on the preset log type to obtain the fragment log with the determined log type.

IN the embodiment of the invention, the application server contains all input and output data of the host system, and the log type setting is carried OUT on the service application data to be processed according to the calling service application data IN the host system and the calling service application data outside the system, for example, the log type setting is preset to be a DAUDIT _ OUT type of a component output interaction IN the system, an STAT _ IN type of an interaction input between systems and a STAT _ OUT type of an interaction output between systems.

In the process of implementing step S203 specifically, a message management mechanism in the log analysis component filters a plurality of log data based on a preset log type, so as to determine a log type corresponding to each fragmented log.

Step S204: and processing the fragment logs of the corresponding types according to the processing modes of the fragment logs of different log types to obtain a processing result.

In the process of implementing step S204 specifically, the message management mechanism processes the fragmented logs in sequence according to the log types of the fragmented logs and the processing modes of the fragmented logs corresponding to the log types, so as to obtain processing results, and temporarily stores the processing results in the temporary storage mechanism.

In the embodiment of the invention, a service application log to be processed is obtained according to a first preset time interval; carrying out fragmentation processing on the service application log to obtain a plurality of fragment data; screening a plurality of fragment data based on a preset log type to obtain a fragment log with a determined log type; and processing the fragment logs of the corresponding types according to the processing modes of the fragment logs of different log types to obtain a processing result. In the embodiment of the invention, the log type of each fragmented log is determined through the preset log type, and the fragmented logs of different log types are processed to determine the fragmented logs capable of constructing the loopback data in the fragmented logs of a plurality of log types, so that the completeness and the accuracy of the log data in the process of processing the service application logs are ensured.

Based on the above-described service application log processing method, in the process of specifically executing step S203 to filter a plurality of fragment data based on a preset log type to obtain a fragment log with a determined log type, as shown in fig. 3, the method includes the following steps:

step S301: and for each piece of fragment data, reading the fragment log corresponding to the log offset in the internal cache according to the log offset in the piece of fragment data.

In the process of implementing step S301 specifically, the message management mechanism reads the shard log in the internal cache of the Spark cluster, that is, the temporary storage mechanism, according to the log start offset in the first shard data.

Step S302: and filtering invalid fragment logs in the plurality of fragment logs based on preset and stored server names and monitoring indexes to obtain valid fragment logs.

In step S302, the message management entity filters the fragmentation log that does not include the preset stored server name and the monitoring index, that is, the invalid fragmentation log, to obtain the valid fragmentation log.

It should be noted that the monitoring index refers to data for evaluating whether the service application itself has a problem.

Step S303: and screening effective fragmented logs based on the preset log type to obtain fragmented logs with the log type being the preset log type.

IN step S303, the preset log types include an intra-system component output interaction DAUDIT _ OUT type, an inter-system interaction input STAT _ IN type, and an inter-system interaction output STAT _ OUT type.

In the process of implementing step S303 specifically, the valid fragmented logs are identified according to the preset log type, so as to obtain fragmented logs with the log type being the preset log type.

It should be noted that the fragment log of the DAUDIT _ OUT type includes user identity information of a request sent to a service application, and specifically includes information such as a user ID, an operation ID, an area ID, a system name, and an ID of an organization to which the user corresponds; the STAT _ IN type fragment log comprises real sending request information, a transaction ID and a user ID; STAT OUT type log contains true reply response information, transaction ID and user ID.

Optionally, each fragmented log may include information that affects the processing of the fragmented log, such as noise, and therefore, after the effective fragmented log is obtained by screening, data cleaning may be performed on the effective fragmented log to remove the information that affects the processing of the fragmented log in the effective fragmented log.

In the embodiment of the invention, aiming at each fragment data, reading a fragment log corresponding to the log offset in an internal cache according to the log offset in the fragment data; filtering invalid fragment logs in the plurality of fragment logs based on preset and stored server names and monitoring indexes to obtain valid fragment logs; and screening effective fragmented logs based on the preset log type to obtain fragmented logs with the log type being the preset log type. And processing the fragment logs of the corresponding types according to the processing modes of the fragment logs of different log types to obtain processing results. In the scheme, the log type of each fragmented log is determined through the preset log type, and fragmented logs of different log types are processed to determine fragmented logs capable of constructing loop data in the fragmented logs of multiple log types, so that the completeness and accuracy of log data in the process of processing the service application logs are ensured.

Based on the above-described service application log processing method, in the process of specifically executing step S204 to process the fragmented logs of corresponding types according to the processing manners of the fragmented logs of different log types, and obtain the processing result, as shown in fig. 4, the method includes the following steps:

step S401: judging whether the screened fragment logs have fragment logs of three types including a DAUDIT _ OUT type, a STAT _ IN type and a STAT _ OUT type, if only one or two types of fragment logs of the DAUDIT _ OUT type, the STAT _ IN type and the STAT _ OUT type exist, executing the step S402, and if the screened fragment logs have fragment logs of three types including the DAUDIT _ OUT type, the STAT _ IN type and the STAT _ OUT type, executing the steps S403 to S408.

IN the process of specifically implementing step S401, determining a log type of the screened valid log, if the log type of the screened valid log only includes one or two types of fragmented logs of a DAUDIT _ OUT type, a STAT _ IN type, and a STAT _ OUT type, executing step S402, and if the log type of the screened valid log includes three types of fragmented logs of a DAUDIT _ OUT type, a STAT _ IN type, and a STAT _ OUT type, executing steps S403 to S408.

Step S402: the service application log is discarded.

In the process of implementing step S402 specifically, the service application log obtained in the first preset time interval is discarded.

Step S403: and searching the transaction ID of the fragment log of the DAUDIT _ OUT type aiming at the fragment log of each DAUDIT _ OUT type.

IN the embodiment of the present invention, multiple fragment logs of the DAUDIT _ OUT type, fragment logs of the STAT _ IN type, and fragment logs of the STAT _ OUT type may exist IN the log types of the effective logs obtained by screening. Therefore, the valid fragment logs corresponding to the log types need to be processed in sequence.

In the process of implementing step S403 specifically, for a fragment log of the to-be-processed DAUDIT _ OUT type, a transaction ID in the fragment log of the to-be-processed DAUDIT _ OUT type is searched.

Step S404: searching whether the STAT _ IN type fragment log and the STAT _ OUT type fragment log containing the transaction ID exist IN all effective fragment logs within preset time according to the transaction ID, if the STAT _ IN type fragment log does not exist, executing step S405, if the STAT _ OUT type fragment log does not exist, executing step S406, and if both the STAT _ IN type fragment log and the STAT _ OUT type fragment log exist, executing step S407 and step S408.

IN the process of specifically implementing the step S404, whether all the STAT _ IN type segment logs and the STAT _ OUT type segment logs contain the transaction ID is searched within a preset time, and if the STAT _ OUT type segment logs containing the transaction ID consistent with the transaction ID are not searched, the step S405 is executed; if the STAT _ IN type fragment log containing the transaction ID consistent with the transaction ID cannot be found, executing step S406; if the STAT _ IN type segment log and the STAT _ OUT type segment log containing the transaction ID that matches the transaction ID are found, step S407 and step S408 are executed.

It should be noted that the preset time is set through a plurality of experiments, and may also be set according to the experience of the technician, for example, the preset time may be set to 5 seconds.

Step S405: the fragmentation log of DAUDIT _ OUT type is discarded.

In the embodiment of the present invention, when it is determined that the STAT _ OUT type fragmentation log does not exist in the service application log, it indicates that the DAUDIT _ OUT type fragmentation log is not data that forms a complete loop, and therefore, in the process of implementing step S405 specifically, the DAUDIT _ OUT type fragmentation log corresponding to the transaction ID is discarded.

Step S406: and determining to record the log record corresponding to the fragment log of the DAUDIT _ OUT type as a timeout record.

In the process of implementing step S406 specifically, it is determined to record the log record corresponding to the fragment log of the DAUDIT _ OUT type as the timeout record, that is, the processing result corresponding to the fragment log of the DAUDIT _ OUT type, and the timeout record is temporarily stored.

Step S407: and forming initial loop data by using the DAUDIT _ OUT type fragment log, the STAT _ IN type fragment log and the STAT _ OUT type fragment log which contain the same transaction ID.

IN the process of the specific implementation step S407, because the STAT _ IN type fragment log contains real sending request information, the STAT _ OUT type fragment log contains real reply information, and the DAUDIT _ OUT type fragment log contains user identity information of the service application sending request, the DAUDIT _ OUT type fragment log, the STAT _ IN type fragment log, and the STAT _ OUT type fragment log containing the same transaction ID can form a complete initial loop data, so that the service application logs are contextually responded to each other, thereby forming a complete loop data structure.

Step S408: and processing the initial loop data to obtain a processing result.

In the embodiment of the present invention, since the mainframe system may have an asynchronous processing mechanism that once sends a request message and replies a response message for many times, the initial loopback data needs to be further processed to obtain a processing result.

IN the embodiment of the invention, the log type of each fragment log IN the service application log to be processed is determined, the transaction ID of each fragment log of the DAUDIT _ OUT type is obtained aiming at each fragment log of the DAUDIT _ OUT type, so as to find OUT whether the STAT _ IN type fragment log and the STAT _ OUT type fragment log containing the transaction ID exist or not, if so, an initial loop-back data structure is constructed, and a processing result generated IN the process of constructing the initial loop-back data structure is stored. In the scheme, the log type of each fragmented log is determined through the preset log type, and fragmented logs of different log types are processed to determine fragmented logs capable of constructing loop data in the fragmented logs of multiple log types, so that the completeness and accuracy of log data in the process of processing the service application logs are ensured.

Based on the service application log processing method shown in fig. 4, since the large-scale host system may have an asynchronous processing mechanism for sending the request information once and replying the response information multiple times, the initial loopback data needs to be further processed, and in the process of executing step S408 to process the initial loopback data to obtain the processing result, as shown in fig. 5 in combination with fig. 4, the method includes the following steps:

step S501: determining request information IN initial loop data, if the request information is an asynchronous request, executing the steps S502 to S507, and if the request information is a synchronous request, constructing complete loop data from the DAUDIT _ OUT type fragment log, the STAT _ IN type fragment log and the STAT _ OUT type fragment log.

In step S501, the request information is used to indicate that there is request information for a user to perform an input operation and output a reply in the initial loopback data, and the request information includes a synchronous request or an asynchronous request.

IN the process of specifically implementing step S501, it is determined whether the request information IN the initial loopback data contains an asynchronous identifier, if so, steps S502 to S507 are executed, and if not, it is determined that the request information is a synchronous request, and complete loopback data is constructed from the fragment log of the DAUDIT _ OUT type, the fragment log of the STAT _ IN type, and the fragment log of the STAT _ OUT type.

It should be noted that the synchronous request means that a sending request message is executed after the last sending request message is executed.

Asynchronous request means that when one sending request message is in execution, the next sending request message can start to execute without waiting for the next sending request message to finish execution.

Asynchronous identification refers to a flag used to identify asynchronous requests.

Step S502: and searching whether all the fragment logs contain the fragment logs of other STAT _ IN types which are the same as the user ID or not within preset time according to the user ID IN the STAT _ IN type fragment logs, if the fragment logs of other STAT _ IN types which are the same as the user ID exist, executing a step S503, and if the preset time is exceeded, deleting the STAT _ IN type fragment logs if the fragment logs of other STAT _ IN types which are the same as the user ID do not exist.

IN the process of specifically implementing step S502, because the transaction ID of the reply response information corresponding to the asynchronous request is null, the user ID IN the STAT _ IN type fragment log is obtained, and IN a preset time, whether the user ID exists IN other STAT _ IN type fragment logs IN all fragment logs, if there are other STAT _ IN type fragment logs that are the same as the user ID, step S503 is executed, and if the preset time is exceeded, no other STAT _ IN type fragment log that is the same as the user ID is found, the STAT _ IN type fragment log is deleted.

Step S503: and determining whether asynchronous identifications exist IN the other STAT _ IN type fragment logs, if so, executing step S504, and if not, indicating that multiple times of asynchronous reply response information do not exist.

IN the process of implementing step S503 specifically, it is determined whether an asynchronous identifier also exists IN the other STAT _ IN type fragment logs, if so, it indicates that multiple times of asynchronous reply response information exist, and step S504 is executed, and if not, it indicates that multiple times of asynchronous reply response information do not exist.

Step S504: the preset time is reset.

In the process of implementing step S504 specifically, the preset time is reset.

Step S505: and searching whether a success identifier or a failure identifier exists IN the other STAT _ IN type fragment logs within preset reset time, if so, executing step S506, and if not, executing step S507.

IN the process of specifically implementing step S505, it is determined whether a success identifier or a failure identifier IN other STAT _ IN type shard logs containing the user ID can be found within the preset time of the reset, if the success identifier or the failure identifier can be found, step S506 is executed, and if the success identifier or the failure identifier cannot be found, step S507 is executed.

Step S506: and recording the response time of the searched STAT _ IN type fragment log containing the success identifier or the failure identifier.

IN the process of implementing step S506 specifically, the response time of the found STAT _ IN type segmented log containing the success identifier or the failure identifier, that is, the processing result of processing the STAT _ IN type segmented log is recorded, and the response time is temporarily stored.

It should be noted that the response time is less than or equal to the preset time for resetting.

Step S507: and recording the log record corresponding to the STAT _ IN type fragment log as a timeout record.

IN the process of specifically implementing step S507, it is determined that the STAT _ IN type fragmented log containing the success identifier or the failure identifier cannot be found within the preset time of resetting, which indicates that the lookup is overtime, and records the log record corresponding to the STAT _ IN type fragmented log as an overtime record, that is, records the processing result of processing the STAT _ IN type fragmented log, and temporarily stores the response time.

It should be noted that, after a fragment log of the to-be-processed DAUDIT _ OUT type is processed, the fragment log of the next DAUDIT _ OUT type is continuously searched until the step S303 is executed, until all fragment logs of the DAUDIT _ OUT type in the to-be-processed service application log are circulated, that is, all fragment logs in the to-be-processed service application log are circulated.

IN the embodiment of the invention, the constructed initial loopback data is processed, the request information IN the initial loopback data is determined, and if the request information is an asynchronous request, all the fragment logs are determined to contain the fragment logs of other STAT _ IN types which are the same as the user ID; and determines whether an asynchronous identification exists IN the sharded logs of other STAT _ IN types. When the asynchronous identification exists, if successful identification or failed identification exists IN the other STAT _ IN type fragment logs containing the user ID, and the response time of the searched STAT _ IN type fragment logs containing the successful identification or the failed identification is recorded; and if determining that the success identification or the failure identification does not exist IN the other STAT _ IN type fragmentation logs containing the user ID, recording the log record corresponding to the STAT _ IN type fragmentation log as a timeout record. And storing a processing result generated in the process of processing the initial loop data. In the scheme, the log type of each fragmented log is determined through the preset log type, and fragmented logs of different log types are processed to determine fragmented logs capable of constructing loop data in the fragmented logs of multiple log types, so that the completeness and accuracy of log data in the process of processing the service application logs are ensured.

To better explain the log processing method disclosed in the above embodiment of the present invention, the following examples are given.

Assuming that the first preset time interval is 2 minutes, the designated position is A, the service application log to be processed is B, the preset number of fragments is 10, the preset time is 5 seconds,

the log acquisition component reads the service application log B to be processed at the appointed position A of the application server every 2 minutes by a distributed data acquisition agent technology deployed on the application server, and temporarily stores the acquired service application log B to be processed into a message queue Kafka of the log acquisition component.

The message management mechanism obtains the service application logs to be processed temporarily stored in the message queue Kafka, and fragments the service application logs to be processed according to the preset number of fragments to obtain 10 fragment data.

Each piece of fragment data consists of a fragment log and a log start offset and an end offset of the corresponding fragment log.

And temporarily storing the fragment data into an internal cache of the Spark cluster, and storing the log offset into a message management queue inside the Kafka.

And a message management mechanism in the log analysis component screens 10 log data based on a preset log type, so that the log type corresponding to each fragment log is determined.

The type of the log corresponding to the 1 st fragmented log is a STAT _ OUT type 1, the type of the log corresponding to the 2 nd fragmented log is a STAT _ IN type 1, the type of the log corresponding to the 3 rd fragmented log is a STAT _ IN type 2, the type of the log corresponding to the 4 th fragmented log is a STAT _ OUT type 2, the type of the log corresponding to the 5 th fragmented log is a STAT _ IN type 3, the type of the log corresponding to the 6 th fragmented log is a STAT _ IN type 4, the type of the log corresponding to the 7 th fragmented log is a STAT _ IN type 5, the type of the log corresponding to the 8 th fragmented log is a STAT _ OUT type 3, the type of the log corresponding to the 9 th fragmented log is a STAT _ OUT type 1, and the type of the log corresponding to the 10 th fragmented log is a STAT _ OUT type.

Further, it should be noted that the transaction ID corresponding to the fragment log of DAUDIT _ OUT type 1 is 1234; the transaction ID corresponding to the fragment log of the DAUDIT _ OUT type 2 is 2345; the transaction ID corresponding to the fragment log of the DAUDIT _ OUT type 3 is 3456; transaction IDs corresponding to the slice log of STAT _ IN type 2 and the slice log of STAT _ OUT type 1 are 1234; the transaction IDs corresponding to the fragment logs of STAT _ IN type 1 are 2345; STAT _ OUT type 2 sharded log; the user IDs corresponding to the STAT _ IN type 2 fragment log, the STAT _ IN type 4 fragment log, and the STAT _ IN type 5 fragment log are 11111.

Determining that fragment logs of three types including a DAUDIT _ OUT type, a STAT _ IN type and a STAT _ OUT type exist IN the fragment logs obtained through screening, and acquiring the transaction ID1234 of the fragment log of the DAUDIT _ OUT type 1 aiming at the fragment log of the DAUDIT _ OUT type 1.

The slice log of STAT _ IN type 2 and the slice log of STAT _ OUT type 1 containing the transaction ID1234 are found within 5 seconds.

And (3) forming initial loop data by the fragment log of the DAUDIT _ OUT type 1, the fragment log of the STAT _ IN type 2 and the fragment log of the STAT _ OUT type 1.

Determining that the request information IN the initial loopback data is an asynchronous request, acquiring a user ID11111 IN a STAT _ IN type 2 fragment log, and determining that the STAT _ IN type 4 fragment log and the STAT _ IN type 5 fragment log which are the same as the user ID1111 exist IN 10 fragment logs within 5 seconds.

And determining that asynchronous identifications exist IN the STAT _ IN type 4 fragment log and the STAT _ IN type 5 fragment log.

Resetting the preset time, finding that a success identifier or a failure identifier exists IN the STAT _ IN type 4 fragment log within 5 seconds of resetting, and recording the response time of the found STAT _ IN type 4 fragment log containing the success identifier or the failure identifier as 3 seconds.

And continuing to process the next fragmentation log, namely processing the fragmentation log of the DAUDIT _ OUT type 2 to obtain the transaction ID1234 of the fragmentation log of the DAUDIT _ OUT type 2.

The STAT _ IN type 1 fragmented log containing the transaction ID2345 is found within 5 seconds, and the STAT _ OUT type fragmented log containing the transaction ID2345 is not found.

The fragmentation log of DAUDIT _ OUT type 2 is discarded.

And continuously processing the next fragmentation log, namely processing the fragmentation log of the DAUDIT _ OUT type 3 to obtain the transaction ID3456 of the fragmentation log of the DAUDIT _ OUT type 3.

The STAT _ OUT type 2 fragmented log containing the transaction ID2345 is found within 5 seconds, and the STAT _ IN type fragmented log containing the transaction ID3456 is not found.

And determining to record the log record corresponding to the fragment log of the DAUDIT _ OUT type 3 as a timeout record, and temporarily storing the timeout record into a temporary storage mechanism.

Based on the above-described service application log processing method, with reference to fig. 2, as shown in fig. 6, the method further includes:

step S205: and summarizing the processing result, and calculating an early warning value based on the summarized processing result.

In the process of specifically implementing step S205, each processing result is summarized, and the total amount and/or the average value of each processing result after being summarized is calculated, so as to obtain the early warning value corresponding to each processing result.

Such as: and summarizing each response time recorded in the service application log, and calculating an average value of the summarized response times to obtain an early warning value corresponding to the response time. Or summarizing each overtime record recorded in the service application log, and calculating the times of the overtime records in the service application log to obtain an early warning value corresponding to the overtime record.

Step S206: and judging whether the early warning value is equal to or greater than a set early warning value, and if so, executing the step S207.

It should be noted that the early warning value is set through a plurality of experiments, or may be set by a technician according to experience.

Step S207: and (6) alarming.

In the process of implementing step S207 specifically, an alarm is given by calling an external alarm interface. Such as invoking an external alarm to alarm.

In the embodiment of the invention, the processing results are summarized, the early warning value is calculated based on the summarized processing results, and when the early warning value is determined to be equal to or larger than the set early warning value, an alarm is given. The monitoring alarm component is additionally arranged in the service application log processing system, so that a user can better know the processing condition of the log, and the user can provide certain processing measures conveniently.

Based on the above-described service application log processing method, with reference to fig. 2, as shown in fig. 7, the method further includes:

step S208: and acquiring and displaying a processing result of the service application log to be processed according to a second preset time interval.

In the process of implementing step S208 specifically, the processing result of the service application log to be processed is obtained in real time, and the processing result displayed on the display interface is refreshed according to a second preset time interval.

The second preset time interval is set through a plurality of experiments, and may also be set according to the experience of the technician, for example, it may be set to 10 seconds.

The second preset time interval may be the same as or different from the first preset time interval.

In the embodiment of the invention, the processing result of the service application log to be processed is obtained according to the second preset time interval and displayed. The output display component is additionally arranged in the service application log processing system, so that a user can better know the log processing condition, and the user can provide certain processing measures conveniently.

Based on the service application log processing method shown in the above embodiment of the present invention, an embodiment of the present invention also correspondingly discloses a service application log processing apparatus, as shown in fig. 8, which is a schematic structural diagram of a service application log processing apparatus provided in the embodiment of the present invention, and the service application log processing apparatus includes:

the acquisition module 801 is configured to acquire a service application log to be processed according to a first preset time interval.

The fragmentation module 802 is configured to perform fragmentation processing on the application log to obtain a plurality of fragmentation data.

Optionally, the fragmentation module 802 is specifically configured to: the method comprises the steps that fragmentation processing is carried out on a service application log to be processed, and a plurality of fragmentation logs and log offset of corresponding fragmentation logs are obtained; and temporarily storing the fragment data into an internal cache, and storing the log offset into a message management queue.

It should be noted that each fragment log and the log offset corresponding to the fragment log form fragment data, each fragment log includes log data and redundant data, and the redundant data is used to store a reply packet that is not included in the log data.

The screening module 803 is configured to screen multiple pieces of data based on the log type to obtain a piece of log with a determined log type.

The processing module 804 is configured to process the fragment logs of the corresponding types according to the processing modes of the fragment logs of different log types, so as to obtain a processing result.

It should be noted that, the specific principle and the execution process of each module in the service application log processing apparatus disclosed in the above embodiment of the present invention are the same as the service application log processing method implemented in the above embodiment of the present invention, and reference may be made to the corresponding parts in the service application log processing method disclosed in the above embodiment of the present invention, which are not described herein again.

Based on the service application log processing apparatus shown in the foregoing embodiment of the present invention, the screening module 803 includes:

and the reading unit is used for reading the fragment log corresponding to the log offset in the internal cache according to the log offset in the fragment data aiming at each fragment data.

And the filtering unit is used for filtering invalid fragment logs in the plurality of fragment logs based on the prestored server name and the monitoring index to obtain valid fragment logs.

And the screening unit is used for screening the effective fragmented logs based on the preset log type to obtain the fragmented logs with the log type being the preset log type.

It should be noted that the preset log types include a DAUDIT _ OUT type of intra-system component output interaction, a STAT _ IN type of inter-system interaction input, and a STAT _ OUT type of inter-system interaction output.

Based on the service application log processing apparatus shown in the foregoing embodiment of the present invention, the processing module 804 includes:

and the determining unit is used for discarding the service application log if only one or two types of the DAUDIT _ OUT type, the STAT _ IN type and the STAT _ OUT type exist IN the screened fragment logs.

The first searching unit is used for searching the transaction ID of the fragment log of the DAUDIT _ OUT type aiming at the fragment log of each DAUDIT _ OUT type if fragment logs of three types including the DAUDIT _ OUT type, the STAT _ IN type and the STAT _ OUT type exist IN the fragment logs obtained through screening.

The second searching unit is used for searching whether the STAT _ IN type fragment log and the STAT _ OUT type fragment log containing the transaction ID exist IN all the effective fragment logs within preset time according to the transaction ID, and if the STAT _ OUT type fragment log containing the transaction ID does not exist, the discarding unit is executed; if the STAT _ IN type fragment log containing the transaction ID does not exist, executing a first recording unit; if both exist, the constituent elements are executed.

And the discarding unit is used for discarding the fragmentation log of the DAUDIT _ OUT type.

The first recording unit is used for determining that the log record corresponding to the fragment log of the DAUDIT _ OUT type is recorded as a timeout record.

And the forming unit is used for forming initial loop back data by the DAUDIT _ OUT type fragment log, the STAT _ IN type fragment log and the STAT _ OUT type fragment log which contain the same transaction ID.

And the processing unit is used for processing the initial loop data to obtain a processing result.

Further, based on the service application log processing apparatus shown in the foregoing embodiment of the present invention, the processing unit is specifically configured to: determining request information in initial loop back data; if the request is an asynchronous request, searching whether all the fragment logs contain the fragment logs of other STAT _ IN types which are the same as the user ID or not within preset time according to the user ID IN the fragment logs of the STAT _ IN types; if yes, determining whether asynchronous identifications exist IN other STAT _ IN type fragment logs; if the asynchronous identifier exists, resetting the preset time, and searching whether a successful identifier or a failed identifier exists IN the other STAT _ IN type fragment logs within the reset preset time; if the success identification or the failure identification exists, recording the response time of finding the STAT _ IN type fragment log containing the success identification or the failure identification; and if the success identifier or the failure identifier does not exist, recording the log record corresponding to the STAT _ IN type fragment log as an overtime record.

It should be noted that the request information is used to indicate that there is request information for a user to perform an input operation and output a reply in the initial loopback data, and the request information includes a synchronous request or an asynchronous request.

The service application log processing apparatus shown based on the above embodiment of the present invention further includes:

the alarm module is used for summarizing the processing results and calculating an early warning value based on the summarized processing results; judging whether the early warning value is equal to or greater than a set early warning value; if yes, alarming.

and the display module is used for acquiring and displaying the processing result of the service application log to be processed according to a second preset time interval.

The embodiment of the invention provides electronic equipment, wherein the electronic equipment is used for running a program, and the method for processing the service application log provided by any embodiment of the invention is executed when the program runs.

An embodiment of the present invention provides a computer storage medium, where the computer storage medium includes a storage program, and when the storage program runs, a device on which the storage medium is located is controlled to execute the service application log processing method according to any embodiment of the present invention.

The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the system or system embodiments are substantially similar to the method embodiments and therefore are described in a relatively simple manner, and reference may be made to some of the descriptions of the method embodiments for related points. The above-described system and system embodiments are only illustrative, wherein the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.

Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing service application logs is characterized by comprising the following steps:

2. The method of claim 1, wherein the slicing the application log to obtain a plurality of sliced data comprises:

3. The method of claim 2, wherein the filtering the plurality of sliced data based on a preset log type to obtain a sliced log with a determined log type comprises:

4. The method according to claim 3, wherein the processing the fragmentation logs of the corresponding types according to the processing modes of the fragmentation logs of different log types to obtain a processing result comprises:

and processing the initial loop data to obtain a processing result.

5. The method of claim 4, wherein the processing the initial loopback data to obtain a processing result comprises:

6. The method of claim 1, further comprising:

7. The method of claim 1, further comprising:

8. An apparatus for processing service application logs, the apparatus comprising:

9. An electronic device, characterized in that the electronic device is configured to run a program, wherein the program performs the business application log processing method according to any one of claims 1 to 7 when running.

10. A computer storage medium, characterized in that the storage medium comprises a stored program, wherein the program, when executed, controls an apparatus in which the storage medium is located to perform the service application log processing method according to any one of claims 1 to 7.