CN111782621B

CN111782621B - Business application log processing method and device

Info

Publication number: CN111782621B
Application number: CN202010617619.2A
Authority: CN
Inventors: 刘晓辉; 周凯洋; 李婷; 岳云霞; 于泽
Original assignee: China Travelsky Technology Co Ltd
Current assignee: China Travelsky Technology Co Ltd
Priority date: 2020-06-30
Filing date: 2020-06-30
Publication date: 2023-12-22
Anticipated expiration: 2040-06-30
Also published as: CN111782621A

Abstract

The invention provides a business application log processing method and a device, wherein the method comprises the following steps: acquiring a service application log to be processed according to a first preset time interval; performing slicing processing on the service application log to obtain a plurality of sliced data; screening a plurality of piece of data based on a preset log type to obtain a piece of log of a determined log type; and processing the segmented logs of the corresponding types according to the processing modes of the segmented logs of different log types to obtain processing results. In the embodiment of the invention, the log type of each segmented log is determined through the preset log type, and the segmented logs of the loop data can be constructed in the segmented logs of the plurality of log types through processing the segmented logs of different log types, so that the integrity and the accuracy of the log data in the processing process of the business application log are ensured.

Description

Business application log processing method and device

Technical Field

The present invention relates to the field of big data technologies, and in particular, to a method and an apparatus for processing a service application log.

Background

With the rapid development of civil aviation informatization, an application program running on a large-scale host system generates a large number of logs each day, and the logs record the running state of the host system.

At present, logs are obtained from a message queue one by one based on the time sequence of log generation, and a single log is processed. In the log context processing process, the service application log cannot be directly obtained from the large host system, and the interaction log can be obtained only through the peripheral equipment, so that the sending request and the reply request of the service application exist in different logs, and only the sending request or the reply request is contained in a single log. By processing the log in the mode, the situation that the log context is not matched easily occurs, so that the problems of incomplete data and inaccuracy exist in the processing process of the log.

Disclosure of Invention

In view of the above, the embodiments of the present invention provide a method and an apparatus for processing a service application log, so as to solve the problems in the prior art that, due to unmatched log contexts, there is incomplete data and inaccuracy in the processing process of the log.

In order to achieve the above object, the embodiment of the present invention provides the following technical solutions:

the first aspect of the embodiment of the invention discloses a business application log processing method, which comprises the following steps:

acquiring a service application log to be processed according to a first preset time interval;

Performing slicing processing on the service application log to obtain a plurality of sliced data;

screening the plurality of fragmented data based on a preset log type to obtain a fragmented log with a determined log type;

and processing the segmented logs of the corresponding types according to the processing modes of the segmented logs of different log types to obtain processing results.

Optionally, the performing the slicing processing on the application log to obtain a plurality of sliced data includes:

performing slicing processing on the service application log to be processed to obtain a plurality of slicing logs and log offset corresponding to the slicing logs, wherein each slicing log and the log offset corresponding to the slicing log form slicing data, each slicing log comprises log data and redundant data, and the redundant data is used for storing reply messages which are not contained in the log data;

and temporarily storing the fragment data into an internal cache, and storing the log offset into a message management queue.

Optionally, the screening the plurality of fragmented data based on a preset log type to obtain a fragmented log with a determined log type includes:

for each piece of data, reading a piece of log corresponding to the log offset in an internal cache according to the log offset in the piece of data;

Filtering invalid fragmented logs in the plurality of fragmented logs based on a pre-stored server name and monitoring index to obtain valid fragmented logs;

screening the effective slicing logs based on a preset log type to obtain slicing logs with the log type being the preset log type, wherein the preset log type comprises an intra-system component output interaction DAUDIT_OUT type, an inter-system interaction input STAT_IN type and an inter-system interaction output STAT_OUT type.

Optionally, the processing the corresponding type of the fragmented logs according to the processing mode of the fragmented logs of different log types to obtain a processing result includes:

if only any one or two types of the DAUDIT_OUT type, the STAT_IN type and the STAT_OUT type exist IN the screened segmented logs, discarding the service application log;

if the screened segmented logs have the DAUDIT_OUT type, the STAT_IN type and the STAT_OUT type, searching the transaction ID of the segmented log of the DAUDIT_OUT type according to the segmented log of each DAUDIT_OUT type;

searching whether the STAT_IN type segmented log and the STAT_OUT type segmented log containing the transaction ID exist IN all the effective segmented logs IN a preset time according to the transaction ID;

If the burst log of the STAT_OUT type containing the transaction ID does not exist, discarding the burst log of the DAUDIT_OUT type;

if the burst log of the STAT_IN type containing the transaction ID does not exist, determining that the log record corresponding to the burst log of the DAUDIT_OUT type is recorded as a timeout record;

if the transaction ID exists, the DAUDIT_OUT type fragmented logs containing the same transaction ID, the STAT_IN type fragmented logs and the STAT_OUT type fragmented logs form initial loop data;

and processing the initial loop data to obtain a processing result.

Optionally, the processing the initial loop data to obtain a processing result includes:

determining request information in the initial loop-back data, wherein the request information is used for indicating that a user in the initial loop-back data carries out input operation and outputs replied request information, and the request information comprises synchronous requests or asynchronous requests;

if the request is an asynchronous request, searching whether all the fragmented logs contain fragmented logs of other STAT_IN types which are the same as the user ID IN preset time according to the user ID IN the fragmented logs of the STAT_IN type;

If yes, determining whether asynchronous identification exists IN the fragment logs of the other STAT_IN types;

if the asynchronous identification exists, resetting the preset time, and searching whether the successful identification or the failed identification exists IN the other STAT_IN type fragment logs within the reset preset time;

if the success mark or the failure mark exists, recording the response time of searching the STAT_IN type fragment log containing the success mark or the failure mark;

and if the success identification or the failure identification does not exist, recording the log record corresponding to the STAT_IN type fragmented log as a timeout record.

Optionally, the method further comprises:

summarizing the processing results, and calculating an early warning value based on the summarized processing results;

judging whether the early warning value is equal to or larger than a set early warning value; if yes, alarming.

Optionally, the method further comprises:

and obtaining and displaying the processing result of the service application log to be processed according to a second preset time interval.

The second aspect of the embodiment of the invention discloses a service application log processing device, which comprises:

the acquisition module is used for acquiring a service application log to be processed according to a first preset time interval;

the slicing module is used for slicing the application log to obtain a plurality of sliced data;

The screening module is used for screening the plurality of piece data based on the log type to obtain a piece log of the determined log type;

and the processing module is used for processing the segmented logs of the corresponding types according to the processing modes of the segmented logs of different log types to obtain processing results.

A third aspect of the embodiment of the present invention discloses an electronic device, where the electronic device is configured to execute a program, where the program executes the service application log processing method according to the first aspect of the embodiment of the present invention.

A fourth aspect of the embodiment of the present invention discloses a computer storage medium, where the storage medium includes a storage program, where the program, when executed, controls a device where the storage medium is located to execute the service application log processing method according to the first aspect of the embodiment of the present invention.

Based on the above method and device for processing the service application log provided by the embodiment of the invention, the method comprises the following steps: acquiring a service application log to be processed according to a first preset time interval; performing slicing processing on the service application log to obtain a plurality of sliced data; screening a plurality of piece of data based on a preset log type to obtain a piece of log of a determined log type; and processing the segmented logs of the corresponding types according to the processing modes of the segmented logs of different log types to obtain processing results. In the embodiment of the invention, the log type of each segmented log is determined through the preset log type, and the segmented logs of the loop data can be constructed in the segmented logs of the plurality of log types through processing the segmented logs of different log types, so that the integrity and the accuracy of the log data in the processing process of the business application log are ensured.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of a business application log processing system according to an embodiment of the present invention;

fig. 2 is a flow chart of a method for processing a service application log according to an embodiment of the present invention;

fig. 3 is a schematic flow chart of a slicing process for a service application log according to an embodiment of the present invention;

FIG. 4 is a schematic flow chart of processing a fragmented log with different log types according to an embodiment of the present invention;

FIG. 5 is a schematic flow chart of processing fragmented logs of different log types according to an embodiment of the present invention;

FIG. 6 is a flowchart illustrating another method for processing a service application log according to an embodiment of the present invention;

fig. 7 is a flow chart of another method for processing a service application log according to an embodiment of the present invention;

Fig. 8 is a schematic structural diagram of a service application log processing device according to an embodiment of the present invention.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

In this application, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

In the embodiment of the invention, the log type of each segmented log is determined through the preset log type, and the segmented logs of the loop data can be constructed in the segmented logs of the plurality of log types through processing the segmented logs of different log types, so that the integrity and the accuracy of the log data in the processing process of the business application log are ensured.

Fig. 1 is a schematic diagram of an architecture of a service application log processing system according to an embodiment of the present invention. The business application log processing system comprises: a log acquisition component 11 and a log analysis component 12.

The log analysis component 12 includes a message management mechanism 121 and a temporary storage mechanism 122.

The architecture based on the system realizes the processing procedure aiming at the business application log, which comprises the following steps:

the log collection component 11 collects logs on each service application according to a first preset time interval through a collection technology deployed on an application server, and stores the collected service application logs into a message queue Kafka of the log collection component 11.

The service application log is analyzed and processed by utilizing the Spark environment of the distributed system Hadoop or CDH platform deployed on the log analysis component 12, specifically: the message managing entity 121 acquires the service application log stored in the message queue Kafka in the log acquisition component 11. The slicing process is carried out to obtain a plurality of sliced data, and the sliced data is temporarily stored in an internal cache, namely a temporary storage mechanism 122; screening a plurality of pieces of data according to a preset log type, so that the log type of each piece of log can be determined; and finally, sequentially processing the segmented logs of the corresponding types according to the processing modes of the segmented logs of different log types to obtain processing results, and temporarily storing the processing results in the temporary storage mechanism 122.

With continued reference to fig. 1, the business application log processing system is further provided with a monitoring alarm component 13 and an output display component 14.

The monitoring alarm component 13 monitors the processing result in the temporary storage mechanism 122 in real time, performs summary calculation on the processing result, and alarms when the determined pre-warning value after the summary calculation is equal to or greater than the set pre-warning value.

The output display module 14 acquires the processing result in the temporary storage mechanism 122 and refreshes the display at a second preset time interval.

In the embodiment of the invention, a log acquisition component acquires a service application log to be processed according to a first preset time interval and stores the service application log; the message management mechanism in the log analysis component performs slicing processing on the business application log to obtain a plurality of slicing data, and stores the slicing data in the temporary storage mechanism of the log analysis component; a message management mechanism in the log analysis component screens a plurality of piece of data based on a preset log type to obtain a piece of log of a determined log type; and processing the segmented logs of the corresponding types according to the processing modes of the segmented logs of different log types to obtain processing results. In the embodiment of the invention, the log type of each segmented log is determined through the preset log type, and the segmented logs of the loop data can be constructed in the segmented logs of the plurality of log types through processing the segmented logs of different log types, so that the integrity and the accuracy of the log data in the processing process of the business application log are ensured. Furthermore, the monitoring alarm assembly and the output display assembly are additionally arranged in the service application log processing system, so that a user can better know the processing condition of the log, and a certain processing measure can be provided for the user conveniently.

Based on the above service application log processing system, referring to fig. 2, a flow chart of a service application log processing method provided by an embodiment of the present invention is shown, where the method includes:

step S201: and acquiring a service application log to be processed according to the first preset time interval.

In the specific implementation process of step S201, the log collection component reads the service application log to be processed located at the designated position of the application server according to the first preset time interval by using the distributed data collection agent technology deployed on the application server, and temporarily stores the collected service application log to be processed into the message queue Kafka of the log collection component.

It should be noted that, the first preset time interval is set through multiple experiments, and may also be set according to experience of a technician, for example: can be set to 2 minutes.

Alternatively, the application server is a peripheral device of the mainframe system, and the designated location of the application server records input and output log data of the mainframe system.

Step S202: and performing slicing processing on the service application log to obtain a plurality of sliced data.

In the specific implementation process of step S202, the method includes the following steps:

Step S11: and performing slicing processing on the business application logs to obtain a plurality of slicing logs and log offset corresponding to the slicing logs.

In step S11, each of the fragmented logs and the log offset of the corresponding fragmented log form a fragmented data, and each of the fragmented logs includes log data and redundancy data, where the redundancy data is used to store a reply message that is not included in the log data.

The log offset includes a log start offset and a log end offset. If the segmented log corresponding to the log offset is not the last segmented log, the log ending offset corresponding to the segmented log is the starting offset of the next segmented log.

In the specific implementation process of step S11, the message management mechanism acquires the service application logs to be processed temporarily stored in the message queue Kafka, and fragments the service application logs to be processed according to the preset number of fragments, so as to obtain the fragmented logs corresponding to the preset number of fragments and the log start offset and the log end offset of the corresponding fragmented logs.

It should be noted that the preset number of slices may be set through multiple experiments, and may also be set according to experience of a technician, for example, may be set to 10.

The log offset refers to the distance between the actual address of the fragmented log and the segment address of the segment where it resides.

Step S12: the sliced data is temporarily stored in an internal buffer, and the log offset is stored in a message management queue.

In the specific implementation process of step S12, the sliced data is temporarily stored in an internal buffer of the Spark cluster, that is, the temporary storage mechanism stores the log offset into a message management queue in Kafka.

Alternatively, the log offset may be stored in a message management queue internal to Kafka, a distributed application coordination service Zookeeper, or a high-performance key-value database Redis.

The log offset stored in the message management queue Kafka may be managed by a message management authority, for example: and storing the log ending offset corresponding to the current read segmented log into a message management queue Kafka to serve as the log starting offset of the next segmented log reading.

Optionally, in the process of slicing the service application log to be processed, the message management mechanism in the log analysis component determines whether a request message exists in the log data in each sliced log and the redundant data corresponding to the log data, and a reply message corresponding to the request message, if so, the redundant data in the sliced log is discarded.

Step S203: and screening the plurality of piece of data based on the preset log type to obtain the piece of log with the determined log type.

IN the embodiment of the invention, the application server contains all input and output data of the host system, and the log type setting is carried OUT on the service application data to be processed according to the calling service application data IN the host system and the calling service application data outside the system, for example, the log type setting can be preset as the type of the output interaction DAUDIT_OUT of the components IN the system, the type of the input interaction STAT_IN between the systems and the type of the output STAT_OUT between the systems.

In the specific implementation process of step S203, the message management mechanism in the log analysis component filters the plurality of log data based on the preset log type, so as to determine the log type corresponding to each segmented log.

Step S204: and processing the segmented logs of the corresponding types according to the processing modes of the segmented logs of different log types to obtain processing results.

In the specific implementation process of step S204, according to the log type of the fragmented log, the message management mechanism processes the fragmented log in sequence according to the processing mode of the fragmented log corresponding to the log type, so as to obtain a processing result, and temporarily stores the processing result in the temporary storage mechanism.

In the embodiment of the invention, a service application log to be processed is obtained according to a first preset time interval; performing slicing processing on the service application log to obtain a plurality of sliced data; screening a plurality of piece of data based on a preset log type to obtain a piece of log of a determined log type; and processing the segmented logs of the corresponding types according to the processing modes of the segmented logs of different log types to obtain processing results. In the embodiment of the invention, the log type of each segmented log is determined through the preset log type, and the segmented logs of the loop data can be constructed in the segmented logs of the plurality of log types through processing the segmented logs of different log types, so that the integrity and the accuracy of the log data in the processing process of the business application log are ensured.

Based on the above-mentioned service application log processing method, in the specific execution step S203, a process of screening a plurality of pieces of fragmented data based on a preset log type to obtain a fragmented log of a determined log type, as shown in fig. 3, the method includes the following steps:

step S301: and for each piece of data, reading the piece of log corresponding to the log offset in the internal cache according to the log offset in the piece of data.

In the specific implementation process of step S301, the message management mechanism reads the slice log from the internal cache, i.e. the temporary storage mechanism, of the Spark cluster according to the log start offset in the first slice data.

Step S302: and filtering invalid segmented logs in the plurality of segmented logs based on preset stored server names and monitoring indexes to obtain effective segmented logs.

In step S302, the message management mechanism filters out the fragmented logs that do not include the preset stored server name and monitoring index, i.e. the invalid fragmented logs, to obtain valid fragmented logs.

The monitoring index refers to data for evaluating whether the business application itself has a problem.

Step S303: and screening the effective fragmented logs based on the preset log types to obtain fragmented logs with the log types being the preset log types.

IN step S303, the preset log types include an intra-system component output interaction da udit_out type, an inter-system interaction input stat_in type, and an inter-system interaction output stat_out type.

In the specific implementation process of step S303, the valid fragmented logs are identified according to the preset log type, so as to obtain the fragmented log with the log type being the preset log type.

It should be noted that, the partition log of the daudit_out type includes user identity information of a request sent to the service application, specifically includes information such as a user ID, an operation ID, an area ID, a system name, and an organization ID corresponding to a user sending the request to the service application; the STAT_IN type fragment log comprises real sending request information, transaction ID and user ID; the stat_out type log contains the true reply information, transaction ID, and user ID.

Optionally, because each segmented log may include information that affects the segmented log processing, such as noise, after the effective segmented log is obtained by screening, the effective segmented log may be subjected to data cleaning, so as to remove the information that affects the segmented log processing in the effective segmented log.

In the embodiment of the invention, for each piece of data, reading a piece of log corresponding to the log offset in an internal cache according to the log offset in the piece of data; filtering invalid fragmented logs in the plurality of fragmented logs based on preset stored server names and monitoring indexes to obtain effective fragmented logs; and screening the effective fragmented logs based on the preset log types to obtain fragmented logs with the log types being the preset log types. The method is convenient for processing the corresponding type of segmented logs according to the processing mode of the segmented logs of different log types, and processing results are obtained. In the scheme, the log type of each segmented log is determined through the preset log type, and segmented logs of different log types are processed to determine segmented logs capable of constructing loop data in the segmented logs of the plurality of log types, so that the integrity and the accuracy of log data in the processing process of the business application log are ensured.

Based on the above-mentioned service application log processing method, in the specific execution step S204, the processing method of the corresponding type of fragmented logs according to the processing modes of the fragmented logs of different log types, and in the process of obtaining the processing result, as shown in fig. 4, includes the following steps:

step S401: judging whether the screened segmented logs have the DAUDIT_OUT type, the STAT_IN type and the STAT_OUT type or not, if only the segmented logs of any one or two types of the DAUDIT_OUT type, the STAT_IN type and the STAT_OUT type exist, executing the step S402, and if the screened segmented logs have the segmented logs of the DAUDIT_OUT type, the STAT_IN type and the STAT_OUT type, executing the steps S403 to S408.

IN the specific implementation process of step S401, determining the log type of the screened effective log, if the log type of the screened effective log includes only one or two types of fragmented logs of the daudit_out type, the stat_in type and the stat_out type, executing step S402, and if the log type of the screened effective log includes three types of fragmented logs of the daudit_out type, the stat_in type and the stat_out type, executing steps S403 to S408.

Step S402: the service application log is discarded.

In the specific implementation process of step S402, the service application log acquired in the first preset time interval is discarded.

Step S403: for each partition log of the type Daudit_OUT, the transaction ID of the partition log of the type Daudit_OUT is looked up.

IN the embodiment of the invention, a plurality of partitioned logs of the DAUDIT_OUT type, partitioned logs of the STAT_IN type and partitioned logs of the STAT_OUT type may exist IN the log types of the screened effective logs. Therefore, it is necessary to sequentially process the valid fragmented logs corresponding to the log types.

In the specific implementation process of step S403, for a pending daudit_out type of the partition log, the transaction ID in the pending daudit_out type of the partition log is searched.

Step S404: and searching whether the STAT_IN type fragmented logs containing the transaction ID and the STAT_OUT type fragmented logs exist IN all the effective fragmented logs according to the transaction ID within a preset time, executing step S405 if the STAT_OUT type fragmented logs containing the transaction ID do not exist, executing step S406 if the STAT_IN type fragmented logs containing the transaction ID do not exist, and executing step S407 and step S408 if the STAT_IN type fragmented logs containing the transaction ID do not exist.

IN the specific implementation process of step S404, searching whether all the slice logs of the stat_in type and the slice logs of the stat_out type contain the transaction ID IN a preset time, and if the slice logs of the stat_out type containing the transaction ID consistent with the transaction ID cannot be searched, executing step S405; if the fragment log of stat_in type containing the transaction ID consistent with the transaction ID is not found, step S406 is executed; if the slice log of stat_in type and the slice log of stat_out type containing the transaction ID consistent with the transaction ID are found, step S407 and step S408 are performed.

It should be noted that the preset time may be set by multiple experiments, or may be set according to experience of a technician, for example, may be set to 5 seconds.

Step S405: the fragmented logs of the DAUDIT OUT type are discarded.

In the embodiment of the present invention, when it is determined that the service application log does not have the stat_out type of the fragmented log, it is indicated that the fragmented log of the daudit_out type is not data forming a complete loop, and therefore, in the process of implementing step S405, the fragmented log of the daudit_out type corresponding to the transaction ID is discarded.

Step S406: and determining that the log record corresponding to the partition log of the DAUDIT_OUT type is recorded as a timeout record.

In the specific implementation process of step S406, it is determined that the log record corresponding to the partition log of the daudit_out type is recorded as a timeout record, that is, the processing result corresponding to the partition log of the daudit_out type temporarily stores the timeout record.

Step S407: the fragmented logs of the DAUDIT_OUT type, the fragmented logs of the STAT_IN type and the fragmented logs of the STAT_OUT type containing the same transaction ID form initial loop-back data.

IN the specific implementation process of step S407, the slice log of the stat_in type contains real sending request information, the slice log of the stat_out type contains real reply response information, and the slice log of the daudit_out type contains user identity information of a service application sending request, so that the slice log of the daudit_out type, the slice log of the stat_in type and the slice log of the stat_out type containing the same transaction ID can form a complete initial loop data, so that the service application log context corresponds, and a complete loop data structure is formed.

Step S408: and processing the initial loop data to obtain a processing result.

In the embodiment of the invention, because the large host system may contain an asynchronous processing mechanism for sending the request information once and replying the response information for many times, the initial loop data needs to be further processed to obtain a processing result.

IN the embodiment of the invention, the log type of each segmented log IN the service application log to be processed is determined, and the transaction ID of the segmented log of the DAUDIT_OUT type is acquired for each segmented log of the DAUDIT_OUT type so as to find whether the segmented log of the STAT_IN type and the segmented log of the STAT_OUT type containing the transaction ID exist or not, if so, an initial loop data structure is constructed, and the processing result generated IN the process of constructing the initial loop data structure is stored. In the scheme, the log type of each segmented log is determined through the preset log type, and segmented logs of different log types are processed to determine segmented logs capable of constructing loop data in the segmented logs of the plurality of log types, so that the integrity and the accuracy of log data in the processing process of the business application log are ensured.

Based on the above-mentioned method for processing the service application log shown in fig. 4, since the mainframe system may have an asynchronous processing mechanism for sending the request information once and replying the response information multiple times, further processing is required for the initial loop data, and in the process of executing step S408 to process the initial loop data, the following steps are included in the process of obtaining the processing result, referring to fig. 4, as shown in fig. 5:

Step S501: and determining request information IN the initial loop data, if the request is an asynchronous request, executing the steps S502 to S507, and if the request is a synchronous request, constructing the complete loop data from the DAUDIT_OUT type segmented log, the STAT_IN type segmented log and the STAT_OUT type segmented log.

In step S501, the request information is used to indicate that there is request information for the user to perform input operation and output reply in the initial loop data, where the request information includes a synchronous request or an asynchronous request.

IN the specific implementation process of step S501, it is determined whether the request information IN the initial loop data contains an asynchronous identifier, if so, steps S502 to S507 are executed, if not, it is indicated that the request information is a synchronous request, and the complete loop data is constructed from the daudit_out type of fragmented log, the stat_in type of fragmented log, and the stat_out type of fragmented log.

It should be noted that, the synchronization request refers to that a transmission request message waits for the execution of the previous transmission request message before starting to execute the transmission request message.

Asynchronous requests refer to the fact that one transmission request message is in execution, and the next transmission request message can start to execute the next transmission request message without waiting for the next transmission request message to be executed.

Asynchronous identification refers to a flag used to identify an asynchronous request.

Step S502: and searching whether all the fragmented logs contain other STAT_IN types of fragmented logs which are the same as the user ID IN a preset time according to the user ID IN the STAT_IN types of fragmented logs, if so, executing step S503, and if not, deleting the fragmented logs of the STAT_IN types.

IN the specific implementation process of step S502, because the transaction ID of the reply response message corresponding to the asynchronous request is null, the user ID IN the slice logs of the stat_in type is obtained, and IN the preset time, whether the user ID exists IN the slice logs of other stat_in types IN all the slice logs is searched, if the slice logs of other stat_in types identical to the user ID exist, step S503 is executed, and if the preset time is exceeded, the slice logs of other stat_in types identical to the user ID are not found, and then the slice logs of the stat_in type are deleted.

Step S503: determining whether there is an asynchronous identifier IN the slice logs of other stat_in types, if there is an asynchronous identifier, executing step S504, if there is no asynchronous identifier, indicating that there is no multiple asynchronous reply response messages.

IN the specific implementation process of step S503, it is determined whether an asynchronous identifier exists IN the slice logs of other stat_in types, if so, it indicates that there is multiple asynchronous reply response messages, and step S504 is performed, if not, it indicates that there is no multiple asynchronous reply response messages.

Step S504: reset the preset time.

In the process of concretely implementing step S504, the preset time is reset.

Step S505: and searching whether a success identifier or a failure identifier exists IN the fragment logs of other STAT_IN types within the preset time of the reset, if so, executing the step S506, and if not, executing the step S507.

IN the specific implementation process of step S505, it is determined whether a success identifier or a failure identifier IN the shard log of other stat_in types containing the user ID can be found within the preset time of the reset, if the success identifier or the failure identifier can be found, step S506 is executed, and if the success identifier or the failure identifier cannot be found, step S507 is executed.

Step S506: and recording the response time of the found STAT_IN type of the fragment log containing the success identification or the failure identification.

IN the specific implementation process of step S506, the response time of the found stat_in type of the fragmented log containing the success identifier or the failure identifier is recorded, that is, the processing result of the stat_in type of the fragmented log is processed, and the response time is temporarily stored.

It should be noted that the response time is less than or equal to the preset time for resetting.

Step S507: and recording the log record corresponding to the slice log of the STAT_IN type as a timeout record.

IN the specific implementation process of step S507, if it is determined that the burst log of the stat_in type containing the success identifier or the failure identifier cannot be found within the preset time, the search timeout is described, and the log record corresponding to the burst log of the stat_in type is recorded as the timeout record, that is, the processing result of processing the burst log of the stat_in type is recorded, and the response time is temporarily stored.

It should be noted that, after processing a fragmented log of a type of daudit_out to be processed, searching for a fragmented log of a next type of daudit_out is continued until the execution step S303 is returned to be executed until all fragmented logs of the type of daudit_out in the service application log to be processed are circulated, that is, all fragmented logs in the service application log to be processed are circulated.

IN the embodiment of the invention, the constructed initial loop data is processed, request information IN the initial loop data is determined, and if the request is an asynchronous request, the segmented logs of other STAT_IN types which are the same as the user ID are determined to be contained IN all the segmented logs; and determines whether asynchronous identifications exist IN the sharlog of other stat_in types. When the asynchronous identification exists, if the successful identification or the failed identification exists IN the other STAT_IN type of the fragmented logs containing the user ID, recording the response time of the searched STAT_IN type of the fragmented logs containing the successful identification or the failed identification; if it is determined that the success mark or the failure mark does not exist IN the other STAT_IN type segmented logs containing the user ID, the log record corresponding to the STAT_IN type segmented log is recorded as the overtime record. The processing result generated in the process of processing the initial loop data is stored. In the scheme, the log type of each segmented log is determined through the preset log type, and segmented logs of different log types are processed to determine segmented logs capable of constructing loop data in the segmented logs of the plurality of log types, so that the integrity and the accuracy of log data in the processing process of the business application log are ensured.

In order to better explain the log processing method disclosed in the above embodiment of the present invention, the following description will be given by way of example.

Assuming that the first preset time interval is 2 minutes, the designated position is a, the service application log to be processed is B, the preset number of fragments is 10, the preset time is 5 seconds,

the log acquisition component reads a to-be-processed service application log B at a designated position A of the application server every 2 minutes through a distributed data acquisition agent technology deployed on the application server, and temporarily stores the acquired to-be-processed service application log B into a message queue Kafka of the log acquisition component.

The message management mechanism acquires the service application log to be processed temporarily stored in the message queue Kafka, and fragments the service application log to be processed according to the preset fragment number to obtain 10 fragment data.

Each piece of data consists of a piece of log and a log start offset and a log end offset of the corresponding piece of log.

The sliced data is temporarily stored in an internal cache of the Spark cluster, and the log offset is stored in a message management queue inside Kafka.

And the message management mechanism in the log analysis component filters 10 log data based on the preset log type, so as to determine the corresponding log type of each segmented log.

The 1 st slicing log corresponds to the log type of DAUDIT_OUT type 1, the 2 nd slicing log corresponds to the log type of STAT_IN type 1, the 3 rd slicing log corresponds to the log type of STAT_IN type 2, the 4 th slicing log corresponds to the log type of DAUDIT_OUT type 2, the 5 th slicing log corresponds to the log type of STAT_IN type 3, the 6 th slicing log corresponds to the log type of STAT_IN type 4, the 7 th slicing log corresponds to the log type of STAT_IN type 5, the 8 th slicing log corresponds to the log type of DAUDIT_OUT type 3, the 9 th slicing log corresponds to the log type of STAT_OUT type 1, and the 10 th slicing log corresponds to the log type of STAT_OUT type 2.

Further, it should be noted that, the transaction ID corresponding to the partition log of the daudit_out type 1 is 1234; the transaction ID corresponding to the partition log of DAUDIT_OUT type 2 is 2345; the transaction ID corresponding to the partition log of DAUDIT_OUT type 3 is 3456; the transaction IDs corresponding to the slice logs of STAT_IN type 2 and STAT_OUT type 1 are 1234; the transaction IDs corresponding to the fragment logs of STAT_IN type 1 are 2345; a slice log of stat_out type 2; the user IDs corresponding to the sharded log of stat_in type 2, the sharded log of stat_in type 4, and the sharded log of stat_in type 5 are 11111.

Determining that the screened segmented logs have the DAUDIT_OUT type, the STAT_IN type and the STAT_OUT type, and acquiring the transaction ID1234 of the segmented log of the DAUDIT_OUT type 1 aiming at the segmented log of the DAUDIT_OUT type 1.

The sharded log for stat_in type 2 and the sharded log for stat_out type 1 containing the transaction ID1234 are found within 5 seconds.

The fragmented log of the Daudit_OUT type 1, the fragmented log of the STAT_IN type 2 and the fragmented log of the STAT_OUT type 1 form initial loop-back data.

The request information IN the initial loop data is determined to be an asynchronous request, the user ID11111 IN the slice logs of the STAT_IN type 2 is obtained, and the slice logs of the STAT_IN type 4 and the slice logs of the STAT_IN type 5 which are the same as the user ID1111 exist IN 10 slice logs within 5 seconds.

It is determined that asynchronous identifications exist IN the sharlog of stat_in type 4 and the sharlog of stat_in type 5.

Resetting the preset time, finding that the success mark or the failure mark exists IN the burst log of the STAT_IN type 4 within 5 seconds of resetting, and recording that the response time of the found burst log of the STAT_IN type 4 containing the success mark or the failure mark is 3 seconds.

Processing is continued on the next sharded log, namely processing is performed on the sharded log of the daudit_out type 2, and the transaction ID1234 of the sharded log of the daudit_out type 2 is obtained.

The slice log of stat_in type 1 containing the transaction ID2345 is found within 5 seconds, and the slice log of stat_out type containing the transaction ID2345 is not found.

The partition log of DAUDIT OUT type 2 is discarded.

Processing is continued on the next sharded log, namely processing is performed on the sharded log of the DAUDIT_OUT type 3, and the transaction ID3456 of the sharded log of the DAUDIT_OUT type 3 is obtained.

The slice log of stat_out type 2 containing the transaction ID2345 is found within 5 seconds, and the slice log of stat_in type containing the transaction ID3456 is not found.

And determining that the log record corresponding to the partition log of the DAUDIT_OUT type 3 is recorded as the timeout record, and temporarily storing the timeout record into a temporary storage mechanism.

Based on the above-mentioned service application log processing method, in conjunction with fig. 2, as shown in fig. 6, the method further includes:

step S205: and summarizing the processing results, and calculating an early warning value based on the summarized processing results.

In the specific implementation process of step S205, each processing result is summarized, and the total amount and/or the average value of each processing result after the summarization is calculated, so as to obtain an early warning value corresponding to each processing result.

Such as: and summarizing each response time recorded in the service application log, and calculating the average value of the summarized response time to obtain an early warning value corresponding to the response time. Or summarizing each overtime record recorded in the service application log, and calculating the times of the overtime record in the service application log to obtain an early warning value corresponding to the overtime record.

Step S206: whether the early warning value is equal to or greater than the set early warning value is judged, and if yes, step S207 is executed.

It should be noted that the early warning value is set through multiple experiments, and may also be set by a technician according to experience.

Step S207: and (5) alarming.

In the specific implementation process of step S207, an alarm is given by calling an external alarm interface. Such as invoking an external alarm to alarm.

In the embodiment of the invention, the processing results are summarized, the early warning value is calculated based on the summarized processing results, and the alarm is given when the early warning value is determined to be equal to or larger than the set early warning value. The monitoring alarm assembly is additionally arranged on the service application log processing system, so that a user can better know the processing condition of the log, and a certain processing measure is provided for the user.

Based on the above-mentioned service application log processing method, in conjunction with fig. 2, as shown in fig. 7, the method further includes:

step S208: and obtaining the processing result of the service application log to be processed according to a second preset time interval, and displaying the processing result.

In the specific implementation process of step S208, the processing result of the service application log to be processed is obtained in real time, and the processing result displayed on the display interface is refreshed according to the second preset time interval.

It should be noted that the second preset time interval is set through multiple experiments, and may also be set according to experience of a technician, for example, may be set to 10 seconds.

The second preset time interval may be the same as or different from the first preset time interval.

In the embodiment of the invention, the processing result of the service application log to be processed is obtained according to the second preset time interval and is displayed. The output display component is additionally arranged on the service application log processing system, so that a user can better know the processing condition of the log, and a certain processing measure is provided for the user.

Based on the service application log processing method shown in the above embodiment of the present invention, the embodiment of the present invention correspondingly discloses a service application log processing device, as shown in fig. 8, which is a schematic structural diagram of the service application log processing device provided in the embodiment of the present invention, where the application log processing device includes:

the acquisition module 801 is configured to acquire a service application log to be processed according to a first preset time interval.

The slicing module 802 is configured to perform slicing processing on the application log to obtain a plurality of sliced data.

Optionally, the slicing module 802 is specifically configured to: performing slicing processing on the service application logs to be processed to obtain a plurality of slicing logs and log offset corresponding to the slicing logs; the sliced data is temporarily stored in an internal buffer, and the log offset is stored in a message management queue.

It should be noted that, each segmented log and the log offset of the corresponding segmented log form a piece of segmented data, each segmented log includes log data and redundant data, and the redundant data is used for storing the reply message not included in the log data.

And a screening module 803, configured to screen the plurality of slice data based on the log type, to obtain a slice log of the determined log type.

The processing module 804 is configured to process the fragmented logs of the corresponding types according to the processing modes of the fragmented logs of different log types, so as to obtain a processing result.

It should be noted that, the specific principle and the execution process of each module in the service application log processing device disclosed in the above embodiment of the present invention are the same as those of the service application log processing method implemented in the above embodiment of the present invention, and may refer to the corresponding parts in the service application log processing method disclosed in the above embodiment of the present invention, and will not be described herein again.

Based on the service application log processing device shown in the above embodiment of the present invention, the screening module 803 includes:

and the reading unit is used for reading the segmented log corresponding to the log offset in the internal cache according to the log offset in the segmented data for each segmented data.

And the filtering unit is used for filtering invalid fragmented logs in the plurality of fragmented logs based on the prestored server names and monitoring indexes to obtain effective fragmented logs.

The screening unit is used for screening the effective fragmented logs based on the preset log types to obtain fragmented logs with the log types being the preset log types.

It should be noted that, the preset log types include a system internal component output interaction da udit_out type, an inter-system interaction input stat_in type, and an inter-system interaction output stat_out type.

Based on the service application log processing device shown in the above embodiment of the present invention, the processing module 804 includes:

and the determining unit is used for discarding the service application log if only one or two types of the DAUDIT_OUT type, the STAT_IN type and the STAT_OUT type exist IN the screened segmented log.

The first searching unit is used for searching the transaction ID of the partitioned logs of the DAUDIT_OUT type according to the partitioned log of each DAUDIT_OUT type if the partitioned logs of the DAUDIT_OUT type, the STAT_IN type and the STAT_OUT type exist IN the partitioned logs obtained through screening.

The second searching unit is used for searching whether the STAT_IN type fragmented logs and the STAT_OUT type fragmented logs containing the transaction ID exist IN all the effective fragmented logs IN a preset time according to the transaction ID, and if the STAT_OUT type fragmented logs containing the transaction ID do not exist, the discarding unit is executed; if the STAT_IN type of the partition log containing the transaction ID does not exist, executing a first recording unit; if both exist, the constituent units are executed.

And the discarding unit is used for discarding the partitioned log of the DAUDIT_OUT type.

And the first recording unit is used for determining that the log record corresponding to the fragmented log of the DAUDIT_OUT type is recorded as the overtime record.

The composing unit is used for composing the fragmented logs of the DAUDIT_OUT type, the fragmented logs of the STAT_IN type and the fragmented logs of the STAT_OUT type containing the same transaction ID into initial loop data.

And the processing unit is used for processing the initial loop data to obtain a processing result.

Further, based on the service application log processing device shown in the above embodiment of the present invention, the processing unit is specifically configured to: determining request information in the initial loop data; if the request is an asynchronous request, searching whether all the fragmented logs contain fragmented logs of other STAT_IN types which are the same as the user ID IN preset time according to the user ID IN the fragmented logs of the STAT_IN type; if yes, determining whether asynchronous identification exists IN the fragment logs of other STAT_IN types; if the asynchronous identification exists, resetting the preset time, and searching whether the successful identification or the failed identification exists IN the fragment logs of other STAT_IN types within the reset preset time; if the success mark or the failure mark exists, recording the response time of searching the STAT_IN type fragment log containing the success mark or the failure mark; if the success identification or the failure identification does not exist, the log record corresponding to the STAT_IN type fragmented log is recorded as a timeout record.

It should be noted that, the request information is used to indicate that the initial loop data has request information for user input operation and output reply, and the request information includes synchronous request or asynchronous request.

The service application log processing device based on the embodiment of the invention further comprises:

the alarm module is used for summarizing the processing results and calculating an early warning value based on the summarized processing results; judging whether the early warning value is equal to or larger than a set early warning value; if yes, alarming.

and the display module is used for acquiring the processing result of the service application log to be processed according to the second preset time interval and displaying the processing result.

The embodiment of the invention provides an electronic device which is used for running a program, wherein the program executes the business application log processing method provided by any embodiment of the invention when running.

The embodiment of the invention provides a computer storage medium, which comprises a storage program, wherein when the program runs, equipment where the storage medium is controlled to execute the business application log processing method provided by any embodiment of the invention.

In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for a system or system embodiment, since it is substantially similar to a method embodiment, the description is relatively simple, with reference to the description of the method embodiment being made in part. The systems and system embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.

Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative elements and steps are described above generally in terms of functionality in order to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A business application log processing method, the method comprising:

screening the plurality of fragmented data based on a preset log type to obtain a fragmented log with a determined log type, wherein the preset log type comprises an intra-system component output interaction DAUDIT_OUT type, an inter-system interaction input STAT_IN type and an inter-system interaction output STAT_OUT type;

if the transaction ID exists, the DAUDIT_OUT type of the burst log, the STAT_IN type of the burst log and the STAT_OUT type of the burst log which contain the same transaction ID form initial loop data;

and processing the initial loop data to obtain a processing result.

2. The method of claim 1, wherein the slicing the service application log to obtain a plurality of sliced data comprises:

3. The method according to claim 2, wherein the filtering the plurality of fragmented data based on the preset log type to obtain a fragmented log with a determined log type comprises:

and screening the effective slicing logs based on a preset log type to obtain slicing logs with the log type being the preset log type.

4. The method of claim 1, wherein processing the initial loop data to obtain a processing result comprises:

5. The method as recited in claim 1, further comprising:

6. The method as recited in claim 1, further comprising:

7. A business application log processing apparatus, the apparatus comprising:

the screening module is used for screening the plurality of piece data based on a preset log type to obtain a piece log with a determined log type, wherein the preset log type comprises an intra-system component output interaction DAUDIT_OUT type, an inter-system interaction input STAT_IN type and an inter-system interaction output STAT_OUT type;

the processing module is used for discarding the service application log if only one or two types of the DAUDIT_OUT type, the STAT_IN type and the STAT_OUT type exist IN the screened segmented log; if the screened segmented logs have the DAUDIT_OUT type, the STAT_IN type and the STAT_OUT type, searching the transaction ID of the segmented log of the DAUDIT_OUT type according to the segmented log of each DAUDIT_OUT type; searching whether the STAT_IN type segmented log and the STAT_OUT type segmented log containing the transaction ID exist IN all the effective segmented logs IN a preset time according to the transaction ID; if the burst log of the STAT_OUT type containing the transaction ID does not exist, discarding the burst log of the DAUDIT_OUT type; if the burst log of the STAT_IN type containing the transaction ID does not exist, determining that the log record corresponding to the burst log of the DAUDIT_OUT type is recorded as a timeout record; if the transaction ID exists, the DAUDIT_OUT type of the burst log, the STAT_IN type of the burst log and the STAT_OUT type of the burst log which contain the same transaction ID form initial loop data; and processing the initial loop data to obtain a processing result.

8. An electronic device, characterized in that the electronic device is configured to run a program, wherein the program, when run, performs the business application log processing method according to any one of claims 1-6.

9. A computer storage medium, characterized in that the storage medium comprises a stored program, wherein the program, when run, controls a device in which the storage medium is located to perform the service application log processing method according to any one of claims 1-6.